Visual automation builders
Build working flows in Shortcuts, Automator or Power Automate Desktop by arranging actions and variables.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Shortcuts · Automator · Power Automate Desktop
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R01.06.01.01-T01 Quick Action that resizes images for the web
Original instruction
In Shortcuts create a Quick Action 'Resize for Web' (Finder, receives images) that converts each selected image to JPEG at quality 0.8, resizes to 1024 px wide keeping aspect ratio, and saves into ~/Web without asking, overwriting nothing (append -1 on name clash).
- Platforms
mac· macOS- Applications
- Shortcuts
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs `shortcuts run 'Resize for Web' -i` on 6 held-out images (PNG/HEIC, portrait/landscape, a name clash): outputs JPEG, width 1024, aspect ±1 px, in ~/Web, clash renamed. Shortcut must exist in the app DB; Terminal withheld; eslogger detects .shortcut files built outside the editor (shortcuts sign) -> 0.
Proposed partial credit
20% Quick Action exists with image input; 40% correct outputs; 20% location/no prompt; 20% clash handling
R01.06.01.01-T02 Folder Action that files incoming PDFs
Original instruction
With Automator create a Folder Action attached to ~/Inbox that moves any PDF added there into ~/Docs/PDF, leaving other file types in Inbox. Enable Folder Actions if needed.
- Platforms
mac· macOS- Applications
- Automator
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
drag· Drag & dropclick· Point & click- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
~/Library/Workflows/Applications/Folder Actions/*.workflow exists with Filter Finder Items (PDF) + Move Finder Items; Folder Actions enabled and bound to ~/Inbox; grader drops 2 PDFs + 1 PNG: PDFs moved within 10 s, PNG stays. Terminal withheld; eslogger detects .workflow/.app bundles written by any process other than Automator -> 0.
Proposed partial credit
30% workflow saved; 20% attached and enabled; 50% behavior on test drop
R01.06.01.01-T03 Power Automate Desktop flow that creates vendor folders
Original instruction
Build a Power Automate Desktop flow 'Vendor Folders' that reads column 'Vendor' from C:\fixtures\invoices.xlsx, and creates one folder per distinct vendor under C:\Vendors (trimmed, case-insensitive dedupe), skipping folders that already exist.
- Platforms
win· Windows- Applications
- Power Automate Desktop · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs the saved flow (PAD console 'Run' via UI automation harness) on a held-out invoices.xlsx: folder set equals distinct trimmed vendors; pre-existing folder untouched. Flow definition must be the PAD designer's saved flow. No Dataverse/API token is issued and PowerShell is AppLocker-blocked, so the flow can only be saved by the PAD designer.
Proposed partial credit
20% flow saved; 30% reads Excel column; 30% correct folder set; 20% dedupe/skip existing
R01.06.01.01-T04 Morning brief shortcut from Calendar and Reminders
Original instruction
Create a shortcut 'Morning Brief' that finds today's Calendar events (all calendars except 'Holidays') and Reminders due today in list 'Work', formats them as a Markdown checklist (time — title), and saves it as a new note 'Brief YYYY-MM-DD' in Notes folder 'Daily'.
- Platforms
mac· macOS- Applications
- Shortcuts · Calendar · Reminders · Notes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droptype· Text & formsclick· Point & click- Work product
WP06· Code/engineering- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Grader seeds events/reminders for a test day, runs the shortcut, then checks the Notes DB: note title with date, folder Daily, lines equal expected ordered list; Holidays events excluded. Terminal withheld; eslogger detects .shortcut files built outside the Shortcuts editor (shortcuts sign/plutil) -> 0.
Proposed partial credit
20% shortcut runs; 30% events filtered/sorted; 25% reminders included; 25% note naming and folder
R01.06.01.01-T05 Automator app that watermarks and combines PDFs
Original instruction
Create an Automator application 'Confidential Pack' on the Desktop: when PDFs are dropped on it, add a 'CONFIDENTIAL' diagonal text watermark, combine them in name order into one PDF named pack_<date>.pdf on the Desktop.
- Platforms
mac· macOS- Applications
- Automator
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
drag· Drag & droptype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader drops 3 PDFs onto the .app (open -a with files): output pages = sum of inputs in name order; each page text layer/overlay contains 'CONFIDENTIAL'; filename date format; workflow saved as Automator application bundle. Terminal withheld; eslogger detects .workflow/.app bundles written by any process other than Automator -> 0.
Proposed partial credit
20% app saved; 30% watermark on all pages; 30% combined in order; 20% naming
R01.06.01.01-T06 Web report download flow with failure handling
Original instruction
Build a PAD flow that opens http://portal.test in Edge, logs in with credentials from the PAD credential store, downloads the 'Monthly report' CSV, saves it as C:\Reports\<yyyy-MM>\report.csv, and on login failure shows a message 'Portal login failed' and stops.
- Platforms
win· Windows- Applications
- Power Automate Desktop · Edge
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droptype· Text & formsclick· Point & click- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
Grader runs the flow twice: valid credentials → file at dated path equals server CSV; credentials rotated → message box text exact and no file. Flow inspected for On block error/If on login result; plain-text password in flow variables -> 0 for that milestone. No Dataverse/API token is issued and PowerShell is AppLocker-blocked, so the flow can only be saved by the PAD designer.
Proposed partial credit
20% browser login steps; 30% download saved correctly; 30% failure branch; 20% credentials not hardcoded
R01.06.01.01-T07 Excel-to-legacy-app data entry flow with UI selectors
Original instruction
Build a PAD flow that, for each of the rows in C:\fixtures\orders.xlsx, fills the 7-field 'New Order' form of the fixture WinForms app OrderDesk (no API) using UI element actions, clicks Save, and writes the returned order number back to column H.
- Platforms
win· Windows- Applications
- Power Automate Desktop · Excel · OrderDesk
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro applong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader runs flow on a held-out 25-row sheet: OrderDesk DB has 25 orders with exact field values; column H equals assigned numbers; flow uses UI element selectors (not image coordinates) per flow definition. No Dataverse/API token is issued and PowerShell is AppLocker-blocked, so the flow can only be saved by the PAD designer.
Proposed partial credit
20% loop over rows; 40% field accuracy; 20% order numbers written back; 20% robust selectors
R01.06.01.01-T08 Low-battery personal automation on iOS
Original instruction
On the iOS simulator create a Personal Automation in Shortcuts: when battery level falls below 20%, turn on Low Power Mode and set brightness to 30%, running immediately without asking.
- Platforms
ios· iOS/iPadOS- Applications
- Shortcuts
- Interfaces
mob· Mobile touch- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
touch· Touch gesturestype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Simulator Shortcuts DB has an automation with trigger battery level <20 falling, actions Set Low Power Mode on + Set Brightness 0.3, runAfterConfirmation=false; grader simulates battery 19% and checks Low Power Mode on. Host Terminal withheld; eslogger on the simulator host detects simctl or Shortcuts DB edits -> 0.
Proposed partial credit
30% trigger; 40% two actions; 20% run immediately; 10% fires in simulation
R01.06.01.01-T09 Water-intake logger that asks for input
Original instruction
Create a shortcut 'Log Water' that asks for an amount in ml (number input, default 250), appends a line 'ISO-timestamp,amount' to iCloud Drive/Shortcuts/water.csv (creating it with header 'time,ml' if missing), and shows today's total.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Shortcuts
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droptype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs with inputs 250, 400 (stdin/answers injected): water.csv has header + 2 rows with valid timestamps; final notification text shows 650; Ask for Input action type Number with default 250. Terminal withheld; eslogger detects .shortcut files built outside the Shortcuts editor (shortcuts sign/plutil) -> 0.
Proposed partial credit
25% ask-for-input config; 35% CSV append incl. header creation; 25% total computed; 15% runs cleanly twice
R01.06.01.01-T10 Title-case Quick Action for selected text
Original instruction
In Automator create a Quick Action 'Title Case' that receives selected text in any application, changes it to Title Case, and replaces the selected text (output replaces selected text).
- Platforms
mac· macOS- Applications
- Automator
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & click- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
~/Library/Services/Title Case.workflow: input type text, any app, 'Output replaces selected text' on, Change Case action Title Case; grader selects text in TextEdit, invokes service, and checks replacement. Terminal withheld; eslogger detects .workflow/.app bundles written by any process other than Automator -> 0.
Proposed partial credit
30% workflow saved as Quick Action; 30% correct action; 40% replaces text in live test
R01.06.01.01-T11 Save vendor attachments from Outlook into dated folders
Original instruction
Build a PAD flow that reads unread Outlook emails from senders @vendor.test, saves their PDF attachments into C:\AP\<yyyy-MM-dd of received>\, marks each processed email read and categorizes it 'Filed', leaving other emails untouched.
- Platforms
win· Windows- Applications
- Power Automate Desktop · Outlook
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droptype· Text & formsclick· Point & click- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro appmultiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Grader seeds a mailbox (12 vendor emails, decoys from vendor.test.evil and @other.test), runs the flow: saved PDFs equal expected set by date folder; processed emails read+category Filed; decoys untouched. No Graph/EWS credentials or Dataverse token issued and PowerShell AppLocker-blocked, so the flow can only be saved by the PAD designer.
Proposed partial credit
20% Outlook retrieve filter; 40% attachments saved correctly; 25% mark/categorize; 15% decoys untouched
R01.06.01.01-T12 Shortcut that files screenshots by month
Original instruction
Create a shortcut 'File Screenshots' that finds files on the Desktop whose names start with 'Screenshot', renames each to YYYYMMDD-HHmmss.png from its creation date, and moves it into ~/Screenshots/YYYY-MM/ (creating folders as needed).
- Platforms
mac· macOS- Applications
- Shortcuts
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droptype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader seeds 10 screenshots across 3 months + 2 decoy files, runs the shortcut: destination paths/names equal expected; decoys stay; no data loss (hashes). Terminal withheld; eslogger detects .shortcut files built outside the Shortcuts editor (shortcuts sign/plutil) -> 0.
Proposed partial credit
20% filter correct; 40% rename format; 30% month folders; 10% decoys untouched
R01.06.01.01-T13 Merge open rows from many CSVs into one workbook
Original instruction
Build a PAD flow that loops through all CSV files in C:\fixtures\tickets\, keeps rows where Status = 'Open', adds a Source column with the file name, and writes them into a new Excel workbook C:\out\open_tickets.xlsx sorted by Priority then Created.
- Platforms
win· Windows- Applications
- Power Automate Desktop · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droptype· Text & forms- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs on a held-out folder (7 CSVs, one with BOM and one empty): output rows equal expected filtered set with Source column, ordered correctly; flow uses loop + data table actions. No Dataverse/API token is issued and PowerShell is AppLocker-blocked, so the flow can only be saved by the PAD designer.
Proposed partial credit
25% loop over files; 30% filter; 15% Source column; 30% sort and Excel output
R01.06.01.01-T14 Recreate a shortcut from a screenshot of its actions
Original instruction
input/shortcut.png shows a 9-action shortcut 'Meeting Notes' (Choose from Menu with 2 branches, Get Upcoming Events, Text with variables, Create Note). Recreate it exactly: same actions, order, parameters and variable wiring.
- Platforms
mac· macOS- Applications
- Shortcuts
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentdrag· Drag & droptype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Export the shortcut plist (grader) and compare action identifiers, order, parameter values and variable references to the reference graph; per-action credit. Behavioral run on seeded calendar must produce the same note as reference. Terminal withheld; eslogger detects .shortcut files built outside the Shortcuts editor (shortcuts sign/plutil) -> 0.
Proposed partial credit
50% actions/order; 30% parameters/variables; 20% behavioral equivalence
R01.06.01.01-T15 Weekly archive via an Automator Calendar Alarm
Original instruction
Create an Automator Calendar Alarm that zips ~/Reports into ~/Archive/reports-<date>.zip, and attach it to a recurring Calendar event every Friday at 17:00 in calendar 'Automation'.
- Platforms
mac· macOS- Applications
- Automator · Calendar
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Calendar DB has weekly Friday 17:00 event with an open-file alarm pointing at the saved .app; grader runs the alarm app: zip created with correct name and contents hash manifest. Terminal withheld; eslogger detects bundles not written by Automator, or zip/ditto run by the agent -> 0.
Proposed partial credit
30% Calendar Alarm workflow saved; 35% recurring event with alarm; 35% archive output correct
R01.06.01.01-T16 Repair a flow broken by a UI update
Original instruction
PAD flow 'Daily Upload' fails since the fixture app UploadDesk was updated (renamed buttons, new confirmation dialog). Fix its UI element selectors and add handling for the new dialog so the flow completes on the updated app.
- Platforms
win· Windows- Applications
- Power Automate Desktop · UploadDesk
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
click· Point & clickdrag· Drag & dropvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
Grader runs the repaired flow 3 times on the updated app: all 3 uploads recorded in UploadDesk DB; dialog handled each time; flow diff shows selector edits and new dialog step, other actions unchanged. PowerShell blocked by AppLocker and no Dataverse token issued, so the flow can only be edited in the PAD designer; UploadDesk has no API.
Proposed partial credit
40% selectors repaired; 30% dialog handled; 30% three clean runs
R01.06.01.01-T17 Share-sheet shortcut that saves recipe links to a note
Original instruction
On the iOS simulator create a shortcut 'Save Recipe' that appears in the Share Sheet for URLs and Safari pages, gets the page title, and appends '- [title](url) — date' to the note 'Recipes' in Notes.
- Platforms
ios· iOS/iPadOS- Applications
- Shortcuts · Safari · Notes
- Interfaces
mob· Mobile touch- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
touch· Touch gesturestype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Shortcut definition: share sheet enabled with URL/Safari web page input types; grader shares 2 local recipe pages from Safari: note 'Recipes' gains 2 correctly formatted lines. Host Terminal withheld; eslogger on the simulator host detects simctl or Shortcuts DB edits -> 0.
Proposed partial credit
30% share sheet config; 30% title extraction; 40% note lines appended correctly
R01.06.01.01-T18 Data-entry flow that survives intermittent pop-ups
Original instruction
Build a PAD flow with a main flow and a 'EnterRecord' subflow that types 30 records from C:\fixtures\claims.csv into the fixture app ClaimsDesk. ClaimsDesk randomly shows a 'Server busy — Retry' dialog; the flow must retry up to 3 times per record and log failures to C:\out\failed.csv.
- Platforms
win· Windows- Applications
- Power Automate Desktop · ClaimsDesk
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
drag· Drag & droptype· Text & formsclick· Point & click- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro appinterrupt· Interruptions- Proposed difficulty
- 5 / 5
Proposed verifier
Grader runs with seeded randomness: ClaimsDesk DB has all records that succeeded within 3 retries (exact fields); failed.csv lists exactly the forced-failure records; flow has subflow + retry/error handling blocks. ClaimsDesk has no API; No Dataverse/API token is issued and PowerShell is AppLocker-blocked, so the flow can only be saved by the PAD designer.
Proposed partial credit
20% subflow structure; 40% records entered correctly; 25% retry logic; 15% failure log
R01.06.01.01-T19 Resize-for-web behavior as a shell script (control)
Original instruction
Control pair of T01: write ~/bin/resize_for_web.sh that converts given images to JPEG quality 80, 1024 px wide keeping aspect, into ~/Web with -1 suffix on name clashes, using sips or ImageMagick.
- Platforms
mac· macOS- Applications
- bash · sips
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Same held-out image test as T01; measures GUI-builder vs script gap.
Proposed partial credit
60% correct outputs; 20% location; 20% clash handling
R01.06.01.01-T20 Invoice intake Quick Action into Numbers
Original instruction
Create a Shortcuts Quick Action 'Intake Invoice' for PDFs: extract text, find vendor, invoice number and total with Match Text (regex), append a row to ~/Invoices/ledger.numbers table 'Ledger', move the PDF into ~/Invoices/<vendor>/, and show a notification with the total.
- Platforms
mac· macOS- Applications
- Shortcuts · Numbers · Finder
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droptype· Text & formsclick· Point & click- Work product
WP06· Code/engineering- Conditions
multiapp· Multi-applicensed· Licensed pro app- Proposed difficulty
- 5 / 5
Proposed verifier
Grader runs on 5 held-out invoices from 3 vendors: ledger table gains 5 rows with exact vendor/number/total; PDFs moved to vendor folders; notification text includes totals. Terminal and Script Editor withheld; eslogger detects AppleScript Numbers inserts outside the shortcut -> 0.
Proposed partial credit
25% extraction regexes; 35% ledger rows; 25% filing; 15% notification
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.