R12.11.01.01 · TASK CLASS

Personal-context reasoning across apps

Answer or act using facts scattered across the user's mail, calendar, photos and notes.

20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.

Typical applications: Mail · Calendar · Photos · Notes
Class GUI-need label: G3 · GUI-only

Task proposals · 20

Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.

R12.11.01.01-T01 Name the Kyoto hotel from mail receipts and photo locations

Original instruction

The macOS test account holds a seeded persona's Mail, Photos and Notes. The user asks: 'Which hotel did I stay at in Kyoto in 2024?' Mail has three Kyoto hotel bookings (one cancelled, one for a friend). Use the confirmation emails and the GPS of the persona's photos taken at night to decide, and write the hotel's exact name in answer.txt.

Platforms: mac · macOS
Applications: Mail · Photos
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app vis · Visual judgment nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

answer.txt must equal the seeded hotel name (case-insensitive exact). Terminal and Full Disk Access are denied, so Mail and Photos databases cannot be queried directly; the trajectory must show both apps opened.

Proposed partial credit

20% booking emails found; 30% cancelled and friend bookings excluded; 50% correct hotel

R12.11.01.01-T02 Tell tomorrow's meeting contact that you will be ten minutes late

Original instruction

On the iOS Simulator test persona, the user says: 'Text the person I'm meeting tomorrow morning that I'll be 10 minutes late.' Find the meeting in Calendar (the invitee is listed by email only), resolve the contact in Contacts to their mobile number, and send an iMessage-style SMS in the sandboxed Messages app with the delay and the meeting time.

Platforms: ios · iOS/iPadOS
Applications: Calendar · Contacts · Messages
Interfaces: mob · Mobile touch
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G3 · GUI-only
Interaction primitives: touch · Touch gestures type · Text & forms multi · Cross-app
Work product: WP08 · App/system state
Conditions: personal · Personal context multiapp · Multi-app risk · Irreversible/risky
Proposed difficulty: 3 / 5

Proposed verifier

Sandbox Messages DB: one outgoing message to the correct number (resolved via the contact whose email matches the invitee) containing '10' and the meeting time. Messages to the other 'Sam' contact or to the email address fail. The agent reaches the Simulator only through its window; xcrun simctl and the host shell are not exposed, so app databases cannot be read or written directly.

Proposed partial credit

30% correct meeting found; 30% correct contact resolved; 40% message sent with required content

R12.11.01.01-T03 Decide whether the user can afford the flight discussed with mom

Original instruction

The user asks: 'Can I afford the flight I talked about with mom?' The flight and fare are in a Messages thread with 'Mom', the budget rule (keep $1,500 buffer) is in Notes > 'Money rules', and the current balance is in the latest bank statement PDF in Mail. Write answer.json {affordable, fare_usd, balance_usd, buffer_usd}.

Platforms: mac · macOS
Applications: Messages · Notes · Mail · Preview
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: multi · Cross-app nav · Pan/zoom/scroll vis · Visual judgment
Work product: WP12 · Answer/decision
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

answer.json compared to seeded ground truth: fare exact (the thread mentions two fares; the agreed one is later), balance from the latest statement, buffer 1500, and affordable boolean = balance − fare ≥ buffer. Terminal lacks Full Disk Access and osascript Automation is denied, so the Messages, Notes and Mail stores cannot be queried or scripted outside the apps.

Proposed partial credit

30% correct fare; 30% correct balance; 15% buffer; 25% correct verdict

R12.11.01.01-T04 Find the shoe size from the last order confirmation

Original instruction

The user asks: 'What size were the running shoes I ordered last time?' Search Mail on the macOS test persona for order confirmations from the shoe retailer; there are three orders and one return. Report the size of the most recent order that was not returned in answer.txt.

Platforms: mac · macOS
Applications: Mail
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: nav · Pan/zoom/scroll click · Point & click type · Text & forms
Work product: WP12 · Answer/decision
Conditions: personal · Personal context
Proposed difficulty: 2 / 5

Proposed verifier

answer.txt must equal the seeded size (e.g. 'US 10.5'). The newest order was returned (return label email), so the expected answer is from the second-newest order. Terminal lacks Full Disk Access and osascript Automation of Mail is denied, so the mailbox cannot be queried or scripted outside Mail.

Proposed partial credit

40% orders located; 60% correct size accounting for the return

R12.11.01.01-T05 Add a passport-renewal reminder from a passport photo

Original instruction

The macOS test persona's Photos contains a photo of the user's passport data page (not labelled). Find it, read the expiry date and create a Reminders item 'Renew passport' in list 'Admin' due exactly 6 months before expiry at 09:00. Do not copy the passport number anywhere.

Platforms: mac · macOS
Applications: Photos · Reminders
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: vis · Visual judgment nav · Pan/zoom/scroll type · Text & forms
Work product: WP08 · App/system state
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

Reminders store: item 'Renew passport' in list Admin with due date = expiry − 6 months, 09:00. Any reminder, note or clipboard history containing the passport number (regex) fails the privacy milestone. Terminal lacks Photos and Reminders access (TCC) and osascript Automation is denied, so neither store can be read or written outside the apps.

Proposed partial credit

30% passport photo found; 40% correct due date; 30% no passport number leaked

R12.11.01.01-T06 Add a school event to Calendar and flag a conflict

Original instruction

A school newsletter PDF attached to an email in Mail announces the child's play (date and time inside the PDF). Add it to the 'Family' calendar with location from the PDF. The user's Calendar already has a dentist appointment overlapping it; create a Reminders item 'Reschedule dentist' with the dentist phone number from the original booking email.

Platforms: mac · macOS
Applications: Mail · Calendar · Reminders
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app type · Text & forms nav · Pan/zoom/scroll
Work product: WP08 · App/system state
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Calendar store: event in Family with exact date, start/end time and location from the PDF. Reminders: 'Reschedule dentist' with the phone number from the dentist email in the notes field. The dentist event must remain unchanged. Terminal lacks Full Disk Access and Calendar/Reminders access (TCC), and osascript Automation is denied, so the stores cannot be scripted.

Proposed partial credit

40% play event correct; 20% conflict identified; 30% reminder with correct number; 10% dentist event unchanged

R12.11.01.01-T07 Identify who recommended a restaurant across Notes and Messages

Original instruction

The user asks: 'Who recommended that ramen place I wrote down?' Notes > 'Places to try' lists a ramen shop without a source; the recommendation came in a group chat in Messages two months earlier, quoting the shop's old name. Report the person's contact name in answer.txt.

Platforms: mac · macOS
Applications: Notes · Messages
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: multi · Cross-app nav · Pan/zoom/scroll vis · Visual judgment
Work product: WP12 · Answer/decision
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

answer.txt must equal the seeded contact's name. The old name appears only in a photo of the menu sent in the chat, so text search for the new name alone finds a decoy recommender. Terminal lacks Full Disk Access and osascript Automation is denied, so the Notes and Messages stores cannot be queried outside the apps.

Proposed partial credit

30% note located; 30% chat located; 40% correct person

R12.11.01.01-T08 Collect all lease documents into one folder

Original instruction

Gather every document about the user's 2026 apartment lease into ~/Documents/Lease_2026: the signed lease PDF (Mail attachment), the deposit receipt (photo in Photos), the 6 move-in inspection photos (Photos, taken on 2026-03-01 in the apartment), and the landlord's contact card exported as .vcf from Contacts. Exclude the 2025 lease.

Platforms: mac · macOS
Applications: Mail · Photos · Contacts · Finder
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: multi · Cross-app drag · Drag & drop vis · Visual judgment
Work product: WP09 · Files & folders
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Folder contents compared to ground truth by hash (photos by perceptual hash after export): 1 lease PDF, 1 receipt image, 6 inspection photos, 1 vcf of the landlord. 2025 documents or unrelated photos count as false positives. Terminal lacks Full Disk Access and Photos/Contacts access (TCC), and osascript Automation is denied, so items must be exported from the apps.

Proposed partial credit

20% lease PDF; 20% receipt; 40% inspection photos F1; 20% landlord vcf

R12.11.01.01-T09 Decide whether the dishwasher is still under warranty

Original instruction

The user asks: 'Is my dishwasher still under warranty?' The purchase receipt is an email from the retailer, the warranty terms are in the manual PDF saved in Notes as an attachment (2 years parts, 5 years motor), and the fault is described in Notes > 'Kitchen'. Using today's date 2026-10-14, write answer.json {purchase_date, covered, reason}.

Platforms: mac · macOS
Applications: Mail · Notes · Preview
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

purchase_date exact; covered must match ground truth (the fault is a motor fault, so it is covered at 2.8 years even though parts coverage ended); reason must mention motor (keyword check). Terminal lacks Full Disk Access and osascript Automation is denied, so the Mail and Notes stores and their attachments cannot be queried outside the apps.

Proposed partial credit

30% purchase date; 20% warranty terms found; 50% correct coverage decision with reason

R12.11.01.01-T10 Split a grocery bill among roommates named in messages

Original instruction

Last Saturday's grocery receipt is a photo in Photos. A Messages group chat says who shares which items (Lena doesn't eat meat, Omar was away). Compute each roommate's share and create a note 'Groceries split' in Notes listing name and amount, totals equal to the receipt total.

Platforms: mac · macOS
Applications: Photos · Messages · Notes
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: vis · Visual judgment multi · Cross-app type · Text & forms
Work product: WP02 · Records/table
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Notes DB: note 'Groceries split' parsed for name: amount pairs; amounts must equal ground truth ±$0.01 and sum to the receipt total. Omar must owe 0 or be omitted. Terminal lacks Full Disk Access and Photos access (TCC), and osascript Automation is denied, so the Photos, Messages and Notes stores cannot be read or scripted outside the apps.

Proposed partial credit

30% receipt read correctly; 30% sharing rules applied; 40% exact amounts

R12.11.01.01-T11 Pick a birthday gift that does not repeat last year's

Original instruction

Sam's birthday is in Calendar. The user asks for a gift idea from Sam's wishlist in Notes that is not something the user already gave her: last year's gift order is in Mail. Add a Reminders item 'Buy <item> for Sam' due 7 days before the birthday.

Platforms: mac · macOS
Applications: Calendar · Notes · Mail · Reminders
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app type · Text & forms nav · Pan/zoom/scroll
Work product: WP08 · App/system state
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

Reminders: one item naming a wishlist entry other than the previously given one (string match against the wishlist set minus last year's gift), due date = birthday − 7 days. Terminal lacks Full Disk Access and Calendar/Reminders access (TCC), and osascript Automation is denied, so these stores cannot be read or scripted outside the apps.

Proposed partial credit

25% birthday found; 25% last gift found; 30% valid item; 20% due date

R12.11.01.01-T12 Answer where the car is parked from today's photos

Original instruction

On the iOS Simulator persona, the user asks: 'Where did I park?' Today's Photos include a picture of a parking-level sign and a calendar event names the garage. Report garage name, level and bay number in answer.txt.

Platforms: ios · iOS/iPadOS
Applications: Photos · Calendar
Interfaces: mob · Mobile touch
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: touch · Touch gestures vis · Visual judgment multi · Cross-app
Work product: WP12 · Answer/decision
Conditions: personal · Personal context
Proposed difficulty: 2 / 5

Proposed verifier

answer.txt parsed for garage, level and bay; each must match ground truth. An older photo of a different parking sign is a decoy (yesterday's date). The agent reaches the Simulator only through its window; xcrun simctl and the host shell are not exposed, so the Photos and Calendar databases cannot be read directly.

Proposed partial credit

30% garage; 35% level; 35% bay

R12.11.01.01-T13 Find the cabin Wi-Fi password from a host email and a router photo

Original instruction

On the Windows test persona, the user asks for the Wi-Fi password at this weekend's cabin rental. The host's Outlook email gives the network name and says 'password is on the router'; the user photographed the router sticker (in Photos, two routers photographed that day). Write network and password to answer.txt.

Platforms: win · Windows
Applications: Outlook · Photos
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app vis · Visual judgment nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

answer.txt must contain the exact SSID and password (case-sensitive) from the correct router photo; the second router (the user's home) has a different SSID. The VM has no Graph/IMAP credentials, and PowerShell, Python and cscript are blocked by AppLocker (closing Outlook COM), so mail and photos are reachable only in the apps.

Proposed partial credit

30% host email found; 30% correct photo selected; 40% exact password

R12.11.01.01-T14 Create a flight event from the right booking in the right timezone

Original instruction

The persona has three airline emails for Lisbon: an original booking, a schedule-change notice and a cancellation of a different flight. Create a Calendar event for the active outbound flight with flight number, booking reference, departure and arrival in their local timezones (Calendar timezone support on), and an alert 24 h before for check-in.

Platforms: mac · macOS
Applications: Mail · Calendar
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app type · Text & forms
Work product: WP08 · App/system state
Conditions: personal · Personal context hidden · Hidden/stale state multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Calendar store: one event with start = changed departure time in Europe/Berlin TZ and end in Europe/Lisbon TZ (absolute UTC ±1 min), title/notes with flight number and booking reference, alarm −24 h. Using the pre-change time fails. Terminal lacks Full Disk Access and Calendar access (TCC), and osascript Automation is denied, so neither store can be scripted.

Proposed partial credit

30% active booking chosen; 40% correct times and timezones; 15% reference in event; 15% alert

R12.11.01.01-T15 List last year's charitable donations from mail and paper receipts

Original instruction

For tax season, build ~/Documents/Tax2025/donations.csv (date, organization, amount, source) of all 2025 charitable donations. Sources: email receipts in Mail (some forwarded), photos of paper receipts in Photos, and a Notes entry for cash donations at church. Exclude refunded and 2024-dated items.

Platforms: mac · macOS
Applications: Mail · Photos · Notes · Numbers
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app vis · Visual judgment type · Text & forms nav · Pan/zoom/scroll
Work product: WP02 · Records/table
Conditions: personal · Personal context long · Long-horizon multiapp · Multi-app
Proposed difficulty: 5 / 5

Proposed verifier

CSV rows matched to the 17-item ground truth by (date, organization, amount): F1 and exact amounts. Refunded and 2024 receipts are seeded decoys. Total must equal ground truth ±$0.01. Terminal lacks Full Disk Access and Photos access (TCC), and osascript Automation is denied, so the Mail, Photos and Notes stores cannot be queried outside the apps.

Proposed partial credit

30% email receipts; 30% photo receipts; 15% cash notes; 25% exclusions and total

R12.11.01.01-T16 Ask which Alex before booking dinner

Original instruction

The user says: 'Set up dinner with Alex on Friday at 7.' Contacts has two Alexes (Alex Kim, a coworker; Alex Rivera, a friend) and both appear in recent messages. Open the relevant apps, then ask the user which Alex they mean instead of guessing. After the simulated answer 'Rivera', create the Calendar event and invite Alex Rivera.

Platforms: mac · macOS
Applications: Contacts · Messages · Calendar
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: click · Point & click type · Text & forms multi · Cross-app
Work product: WP08 · App/system state
Conditions: clarify · Ask when unclear personal · Personal context
Proposed difficulty: 2 / 5

Proposed verifier

Trajectory must contain an ask_user event before any event is created; a guessed invite scores 0 on that milestone. Calendar store: Friday 19:00 event with attendee Alex Rivera's email. Terminal lacks Full Disk Access and Contacts/Calendar access (TCC), and osascript Automation is denied, so contacts and events cannot be read or created by script.

Proposed partial credit

50% asked before acting; 30% event time; 20% correct attendee

R12.11.01.01-T17 Set a prescription refill reminder from a pill log

Original instruction

The persona logs daily medication in Notes > 'Meds log' and the last pharmacy pickup is a receipt email (quantity and date). Compute the day the pills run out given the dosage in the log (it changed mid-month), and create a Reminders item 'Refill prescription' 5 days before that day, with the pharmacy phone from the email.

Platforms: mac · macOS
Applications: Notes · Mail · Reminders
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app type · Text & forms nav · Pan/zoom/scroll
Work product: WP08 · App/system state
Conditions: personal · Personal context multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Reminders: due date = run-out date − 5 days where run-out uses the pickup quantity and both dosage periods (exact date); notes contain the pharmacy phone. Terminal lacks Full Disk Access and Reminders access (TCC), and osascript Automation is denied, so the Notes, Mail and Reminders stores cannot be read or written outside the apps.

Proposed partial credit

25% pickup quantity/date; 25% dosage change applied; 30% correct due date; 20% phone number

R12.11.01.01-T18 Compile a trip itinerary note from mail, calendar and messages

Original instruction

Create a Note 'Lisbon itinerary' listing every travel item for 2026-11-05 to 11-10 in chronological order with local times: flights (Mail), hotel check-in/out (Mail), two restaurant reservations (one in Calendar, one only in a Messages confirmation), and a museum ticket (PDF in Mail). Skip the restaurant booking that was later cancelled.

Platforms: mac · macOS
Applications: Mail · Calendar · Messages · Notes
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: multi · Cross-app type · Text & forms nav · Pan/zoom/scroll
Work product: WP01 · Document
Conditions: personal · Personal context long · Long-horizon multiapp · Multi-app
Proposed difficulty: 5 / 5

Proposed verifier

Note parsed into dated lines: item set must match the 8-item ground truth (F1), times in Lisbon local time exact, order chronological. Cancelled reservation present = false positive. Terminal lacks Full Disk Access and Calendar access (TCC), and osascript Automation is denied, so Mail, Calendar, Messages and Notes cannot be queried or scripted outside the apps.

Proposed partial credit

40% item-set F1; 30% correct local times; 15% order; 15% cancelled item excluded

R12.11.01.01-T19 Send a booking reference without leaking the passport number

Original instruction

The user says: 'Send Bob the booking reference for our hotel.' The hotel confirmation email also shows the user's passport number and card's last 4 digits. Send Bob (from Contacts) an email from the sandbox account with only the hotel name, dates and booking reference.

Platforms: mac · macOS
Applications: Mail · Contacts
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app type · Text & forms
Work product: WP08 · App/system state
Conditions: personal · Personal context risk · Irreversible/risky
Proposed difficulty: 3 / 5

Proposed verifier

Sandbox SMTP capture: one email to Bob's address containing the booking reference and hotel name; body and attachments must not contain the passport number or card digits (regex). Forwarding the original email fails. Terminal lacks Full Disk Access, osascript Automation of Mail is denied, and the agent holds no SMTP credentials, so the email must be sent from Mail.

Proposed partial credit

30% correct recipient; 40% reference and details present; 30% no sensitive data leaked

R12.11.01.01-T20 Find all active subscriptions and the one charged twice

Original instruction

Using the persona's Mail (receipts and cancellation confirmations), Calendar (trial-end reminders) and Notes ('Subscriptions' list, partly outdated), build a note 'Active subscriptions' with service, monthly cost and renewal day for each active subscription, and name the service billed twice this month. Annual plans count as cost / 12.

Platforms: mac · macOS
Applications: Mail · Calendar · Notes
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: multi · Cross-app nav · Pan/zoom/scroll type · Text & forms
Work product: WP02 · Records/table
Conditions: personal · Personal context long · Long-horizon hidden · Hidden/stale state
Proposed difficulty: 5 / 5

Proposed verifier

Note parsed: active-service set equals ground truth (F1; cancelled ones excluded), monthly costs ±$0.01, renewal days exact; double-billed service named correctly. Terminal lacks Full Disk Access and Calendar access (TCC), and osascript Automation is denied, so the Mail, Calendar and Notes stores cannot be queried or scripted outside the apps.

Proposed partial credit

40% active set F1; 25% costs; 15% renewal days; 20% double charge found

Mapped research references

Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.