Screen-reader & keyboard-only operation
Complete ordinary tasks using only VoiceOver/NVDA/Narrator and the keyboard, as a blind user would.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- VoiceOver · NVDA · Narrator
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R01.09.01.02-T01 Create a grocery note using only VoiceOver and the keyboard
Original instruction
VoiceOver is running on the Mac and the pointer is disabled. Open Notes and create a note titled `Groceries` with a bulleted list of three items: `oat milk`, `rye bread`, `lemons`. Navigate only with VoiceOver commands (VO keys, rotor) and standard keyboard shortcuts.
- Platforms
mac· macOS- Applications
- VoiceOver · Notes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Notes DB (NoteStore.sqlite) has a note titled Groceries with 3 list items in order; input-event tap log contains zero pointer clicks/moves; VoiceOver process active throughout with a non-empty speech log. eslogger detects any osascript/JXA or sqlite3 exec (e.g. writing NoteStore), which voids the task.
Proposed partial credit
30% note created with title; 50% three bullet items; 20% no pointer events
R01.09.01.02-T02 Read the third heading of a page with NVDA browse mode into a file
Original instruction
NVDA is running on the Windows VM with Firefox open on `http://fixtures.local/report.html`. Using NVDA browse-mode heading navigation, find the third level-2 heading and type its exact text into a new Notepad file saved as `C:\Users\agent\Desktop\h3.txt`. Mouse input is disabled.
- Platforms
win· Windows- Applications
- NVDA · Firefox · Notepad
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
h3.txt equals the third H2 text (page has visually misleading styled divs that look like headings but are not); NVDA speech log contains that heading; no pointer events in the input log. Sysmon detects curl/PowerShell/Python fetches of the page, which void the task.
Proposed partial credit
60% correct heading text; 20% file saved at path; 20% keyboard-only verified
R01.09.01.02-T03 Export a Word document to PDF with Narrator and the keyboard only
Original instruction
Narrator is on. Open `C:\Users\agent\Documents\Board_Minutes.docx` in Word, export it as PDF to the Desktop with the name `Board_Minutes_final.pdf`, with 'Document structure tags for accessibility' enabled in the export options. Use only keyboard and Narrator.
- Platforms
win· Windows- Applications
- Narrator · Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP09· Files & folders- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
PDF exists with matching text content and /StructTreeRoot (tagged); Narrator active with speech log; input log has no mouse events. COM automation of Word (process audit) voids the task.
Proposed partial credit
50% PDF exported to correct path/name; 30% tagged PDF; 20% keyboard-only
R01.09.01.02-T04 Find and download a report link using the VoiceOver rotor in Safari
Original instruction
With VoiceOver on, Safari shows `http://fixtures.local/investors.html`, a page with 120 links. Use the VoiceOver rotor's Links list to find the link `Annual report 2025 (PDF)` (not the 2024 or 'summary' variants) and download the file into `~/Downloads`. Pointer use is disabled.
- Platforms
mac· macOS- Applications
- VoiceOver · Safari
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
~/Downloads contains the PDF whose SHA-256 matches annual-2025.pdf (decoys have different hashes); VoiceOver speech log shows the rotor Links menu; zero pointer events. eslogger exec events detect curl/wget or osascript runs (fetching the PDF outside Safari), which void the task.
Proposed partial credit
60% correct file downloaded; 20% rotor used (speech log); 20% keyboard-only
R01.09.01.02-T05 Complete a 3-page benefits form with NVDA where errors are only announced
Original instruction
Using NVDA and Firefox, fill the sandbox benefits application at `http://fixtures.local/benefits/` with data from `C:\input\applicant.json`. Validation errors appear only as aria-live announcements and red borders; the form rejects some date formats. Submit the form and save the confirmation number to `C:\Out\confirmation.txt`.
- Platforms
win· Windowsweb· Any OS (web)- Applications
- NVDA · Firefox
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP08· App/system state- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Sandbox backend has one submission with all 18 fields equal to applicant.json (normalized); confirmation.txt equals the issued number; NVDA speech log contains the aria-live error text at least once; zero pointer events; direct POSTs outside the page (no session CSRF token) are detected in the proxy log and fail.
Proposed partial credit
50% fields correct (scaled); 20% submitted; 15% confirmation saved; 15% keyboard-only
R01.09.01.02-T06 Build a PivotTable in Excel without a mouse
Original instruction
Keyboard only (no screen reader required). In `C:\Users\agent\Documents\sales_2026.xlsx`, create a PivotTable on a new sheet named `Pivot` from table `Sales`, with Region in rows, Quarter in columns, Sum of Revenue as values formatted as currency with no decimals, and grand totals for rows only. Save the workbook.
- Platforms
win· Windows- Applications
- Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP02· Records/table- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
openpyxl/pivotCache XML in the saved file: pivot on sheet Pivot, source Sales, rowFields Region, colFields Quarter, dataField sum Revenue with numFmt currency 0 decimals, colGrandTotals false; input log has zero mouse events; no VBA/COM processes.
Proposed partial credit
25% pivot created on named sheet; 40% field layout; 20% number format; 15% totals
R01.09.01.02-T07 Create a recurring calendar event with VoiceOver
Original instruction
With VoiceOver on and pointer disabled, use Calendar to create an event `Physio` in calendar `Personal`, every Tuesday and Thursday at 08:30-09:15 starting next Tuesday for 8 occurrences, location `Clinic`, with an alert 30 minutes before.
- Platforms
mac· macOS- Applications
- VoiceOver · Calendar
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Calendar store: event Physio in Personal with RRULE FREQ=WEEKLY;BYDAY=TU,TH;COUNT=8, start/end times, location, alarm -PT30M; zero pointer events; VoiceOver active. TCC denies AppleScript/JXA Automation of Calendar and eslogger detects osascript runs or scripted .ics imports, which void the task.
Proposed partial credit
25% event and calendar; 35% recurrence; 20% times/location; 20% alert
R01.09.01.02-T08 Schedule Night light with Narrator in Windows Settings
Original instruction
Narrator is on. In Windows Settings, enable Night light on a custom schedule from 21:00 to 07:00 with strength 60, and turn on 'Show accent color on title bars'. Use Narrator and the keyboard only.
- Platforms
win· Windows- Applications
- Narrator · Windows Settings
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcuts- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
CloudStore bluelightreduction state: schedule enabled, custom hours 21:00-07:00, strength ~60 (±5); DWM ColorPrevalence 1; Narrator speech log present; no pointer events. Sysmon registry events detect CloudStore/DWM writes by reg.exe or PowerShell, voiding the task.
Proposed partial credit
50% schedule exact; 20% strength; 20% accent title bars; 10% keyboard-only
R01.09.01.02-T09 Save an email attachment to a folder using NVDA in Thunderbird
Original instruction
With NVDA and Thunderbird, find the newest message from `invoices@vendor.test` that has an attachment named like `INV-*.pdf`, save the attachment to `C:\Users\agent\Documents\Invoices\2026\`, and move the message to the folder `Processed`. Keyboard and NVDA only.
- Platforms
win· Windows- Applications
- NVDA · Thunderbird
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Target folder contains the attachment with matching hash (newest of 4 similar messages); that message is in Processed (mbox/maildir check), others untouched; zero pointer events. Sysmon detects PowerShell/Python processes reading the Thunderbird profile mbox or writing the Invoices folder, which voids the task.
Proposed partial credit
50% right attachment saved; 30% message moved; 20% keyboard-only
R01.09.01.02-T10 Organise tax files in Finder with VoiceOver
Original instruction
With VoiceOver on, in Finder create folder `~/Documents/Taxes 2025` and move into it the four files in `~/Downloads` whose names contain `2025` and end in `.pdf` (there are decoys from 2024 and `.png` versions). Tag the new folder with the `Red` tag.
- Platforms
mac· macOS- Applications
- VoiceOver · Finder
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Folder exists containing exactly the 4 target files (hash match); decoys remain in Downloads; folder has the Red tag (xattr com.apple.metadata:_kMDItemUserTags); zero pointer events; no mv/cp/xattr/tag commands in the eslogger process audit log (any detected run voids the task).
Proposed partial credit
20% folder; 50% exact file set moved; 15% tag; 15% keyboard-only
R01.09.01.02-T11 Answer a data-table question with NVDA table navigation
Original instruction
Firefox with NVDA shows `http://fixtures.local/fleet.html`, a 40-row table of vehicles. Using NVDA table navigation (Ctrl+Alt+arrows), find the vehicle with the highest `Odometer (km)` among those whose `Depot` is `North` and write its `Plate` to `C:\Out\answer.txt`.
- Platforms
win· Windowsweb· Any OS (web)- Applications
- NVDA · Firefox
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt equals the expected plate; NVDA speech log contains table cell navigation announcements; zero pointer events. Page disables text selection and copy so the table cannot be scraped by select-all. Sysmon also detects curl/PowerShell fetches of the page, which void the task.
Proposed partial credit
70% correct plate; 30% table navigation evidenced in log
R01.09.01.02-T12 Create a Reminders list with VoiceOver gestures on iPhone
Original instruction
On the iOS Simulator with VoiceOver on, use VoiceOver touch gestures (swipe to move, double-tap to activate, rotor twist) to create a Reminders list named `Clinic` with two reminders: `Bring referral letter` and `Fast after 22:00`, the second with a reminder time of tomorrow 21:30.
- Platforms
ios· iOS/iPadOS- Applications
- VoiceOver · Reminders
- Interfaces
mob· Mobile touch- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturestype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Reminders DB: list Clinic with the two reminders, second with due alarm tomorrow 21:30 local; VoiceOver enabled throughout (accessibility log); input log contains VoiceOver gestures (no direct taps on target coordinates while VO focus elsewhere).
Proposed partial credit
25% list created; 40% two reminders; 20% time set; 15% VO gestures used
R01.09.01.02-T13 Set a weekday alarm on Android with TalkBack
Original instruction
On the Android emulator with TalkBack on, open Clock and create an alarm at 06:45 repeating Monday through Friday, labelled `Shift`, with vibration on. Navigate with TalkBack swipe and double-tap gestures.
- Platforms
and· Android- Applications
- TalkBack · Clock
- Interfaces
mob· Mobile touch- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
touch· Touch gestures- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Clock app DB (alarms table) has 06:45, days mask Mon-Fri, label Shift, vibrate 1, enabled; TalkBack enabled during the whole run (settings poll); gesture log shows TalkBack navigation gestures. adb is not exposed to the agent, so the alarm can be set only through the emulator UI.
Proposed partial credit
40% time; 30% repeat days; 15% label; 15% vibrate + TalkBack use
R01.09.01.02-T14 Reorder Kanban cards with keyboard drag-and-drop
Original instruction
In the sandbox Kanban web app (Chrome, keyboard only, no screen reader required), move card `API rate limits` from `Backlog` to the top of `In progress`, and move `Release notes` to directly below `QA sign-off` in `Review`. The app supports keyboard drag via Space to lift and arrows to move.
- Platforms
web· Any OS (web)- Applications
- Chrome · Kanban sandbox
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcuts- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
App DB card positions: API rate limits at In progress index 0; Release notes right after QA sign-off; other cards' relative order unchanged; input log has zero pointer events; API calls only from the page (no REST calls with external tokens).
Proposed partial credit
45% first move; 45% second move; 10% no other order changes
R01.09.01.02-T15 Structure a document with headings and a table of contents via VoiceOver
Original instruction
With VoiceOver on, open `~/Documents/handbook.pages` in Pages. Apply Heading style to the 5 section titles listed in `~/Desktop/sections.txt` and Heading 2 to their 9 subsections, insert a table of contents at the top, and export as PDF to `~/Desktop/handbook.pdf`.
- Platforms
mac· macOS- Applications
- VoiceOver · Pages
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Exported PDF outline/bookmarks match the 5 headings with 9 nested subheadings in order; first page contains TOC entries; Pages file paragraph styles verified via exported .docx copy; zero pointer events. TCC denies AppleScript/JXA Automation of Pages and eslogger detects osascript or textutil runs, which void the task.
Proposed partial credit
40% headings; 25% subheadings; 20% TOC; 15% PDF export
R01.09.01.02-T16 Operate an inaccessible app with NVDA object navigation and OCR
Original instruction
The legacy app `LabelPrinter.exe` has unlabeled icon buttons and a custom-drawn settings panel. With NVDA (object navigation, NVDA OCR) and the keyboard only, set the label size to `62 x 29 mm`, density to `High`, and enable `Cut after each label`, then save the settings.
- Platforms
win· Windows- Applications
- NVDA · LabelPrinter
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
key· Shortcutsvis· Visual judgment- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
App config file (written only on Save) has size 62x29, density High, autocut true; NVDA speech log contains OCR output; zero pointer events. The config file is encrypted, so direct edits fail. Sysmon detects UI-automation scripts (AutoHotkey, pywinauto, PowerShell) driving the app or writing its config, which void the task.
Proposed partial credit
35% label size; 25% density; 25% cut option; 15% saved
R01.09.01.02-T17 Read chart values in Excel with Narrator and log them
Original instruction
With Narrator on, open `C:\Users\agent\Documents\kpi.xlsx`. Sheet `Dashboard` has a column chart built from hidden sheet `Data`. Using Narrator and the keyboard only, find which month had the lowest `Net Promoter Score` and its value, and type `<Month>,<value>` into cell B2 of sheet `Answer`. Save.
- Platforms
win· Windows- Applications
- Narrator · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP12· Answer/decision- Conditions
licensed· Licensed pro apphidden· Hidden/stale state- Proposed difficulty
- 4 / 5
Proposed verifier
Answer!B2 equals expected 'Month,value'; hidden sheet remains hidden; workbook otherwise unchanged (cell diff); zero pointer events; Narrator speech log present. Excel macros are disabled by policy and Sysmon detects PowerShell/Python processes (COM automation, openpyxl) touching kpi.xlsx, voiding the task.
Proposed partial credit
60% correct month+value; 20% hidden sheet left hidden/no other edits; 20% keyboard-only
R01.09.01.02-T18 Complete a sandbox checkout with VoiceOver through a focus-trapped modal
Original instruction
With VoiceOver and Safari, buy one `Braille label tape (12 mm)` from the sandbox store `http://shop.local`. A cookie modal traps focus on load and the shipping form uses custom radio buttons. Use the test card in `~/Desktop/test_card.txt`. Stop at the final 'Place order' review screen and ask for confirmation instead of placing the order.
- Platforms
mac· macOSweb· Any OS (web)- Applications
- VoiceOver · Safari
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 5 / 5
Proposed verifier
Sandbox DB: a pending checkout session with exactly 1 item (right SKU), shipping fields filled, test card tokenized, status 'review' and no placed order; agent message asks for confirmation; zero pointer events; cookie consent recorded as 'necessary only'. The store issues no API token and accepts checkout only from the Safari session (CSRF); eslogger detects curl or osascript runs.
Proposed partial credit
20% cookie modal handled; 30% correct cart; 30% shipping/payment filled; 20% stopped at review
R01.09.01.02-T19 Tell a blind user the keyboard path to Track Changes in Word
Original instruction
Guide mode: a blind user using JAWS-like keyboard navigation asks, 'How do I turn on Track Changes in Word with the keyboard, and how do I hear who made a change?' Explore Word on the VM with Narrator to confirm the key paths, then write the answer to `C:\Out\guide.txt` as numbered steps with exact key sequences.
- Platforms
win· Windows- Applications
- Narrator · Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & forms- Work product
WP01· Document- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
guide.txt checked against ground-truth key paths (Ctrl+Shift+E or Alt,R,G,... and the Reviewing Pane route); rubric graded by LLM against reference (no deterministic alternative for free-text) plus exact match on the shortcut tokens. Word's VBA and COM automation are disabled by policy and the VM has no web access, so key paths can be checked only by operating Word through Narrator.
Proposed partial credit
50% correct toggle key path; 30% reviewer-identification route; 20% clear numbered steps
R01.09.01.02-T20 Move a web table into Numbers and chart it using only VoiceOver
Original instruction
With VoiceOver on, open `http://fixtures.local/rainfall.html` in Safari, copy the monthly rainfall table (12 rows) into a new Numbers spreadsheet, add a column chart of rainfall by month titled `Rainfall 2025`, and save as `~/Documents/rainfall.numbers`. Pointer is disabled.
- Platforms
mac· macOS- Applications
- VoiceOver · Safari · Numbers
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutstype· Text & formsmulti· Cross-app- Work product
WP03· Visual layout- Conditions
multiapp· Multi-applong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Numbers file (exported to CSV/XLSX for checking) has 12 rows matching the source values; one chart of type column referencing them with the exact title; zero pointer events; VoiceOver active in both apps. TCC denies AppleScript/JXA Automation of Safari and Numbers, and eslogger detects curl or osascript runs, which void the task.
Proposed partial credit
40% data transferred; 35% chart type+range; 15% title; 10% saved path
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.