Computer-Use Agent Taxonomy中文
R01.10.01.04 · TASK CLASS

Complex web widgets

Operate date pickers, range sliders, autocomplete, hover menus, drag-and-drop lists and rich selects to exact values.

20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.

Typical applications
Chrome · Safari · Edge
Class GUI-need label
G3 · GUI-only

Task proposals · 20

Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.

R01.10.01.04-T01 Set a dual-thumb price slider to exactly $35-$80

Original instruction

Chrome is open at the replica store http://shop.local/c/backpacks with the price filter at $0-$250 (step $1). Set the dual-thumb slider to exactly $35 (min) and $80 (max) and apply the filter. The min/max text boxes are disabled; use the slider handles with mouse or arrow keys.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · shop.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop geom · Precise geometry
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
2 / 5

Proposed verifier

Request log: last GET /api/products has price_min=35&price_max=80 and URL state matches. Widget records only isTrusted pointer/key events; a programmatic value change scores 0. DevTools and javascript: URLs disabled by Chrome policy.

Proposed partial credit

40% min thumb = 35; 40% max thumb = 80; 20% filter applied (grid refreshed to the expected 14 items)

R01.10.01.04-T02 Pick a December-January stay in a two-month range picker

Original instruction

In Edge, the replica hotel site stay.local opens a two-month range date picker on the current month (October 2026). Select check-in 2026-12-18 and check-out 2027-01-03 and press Search. The date inputs are read-only, so the picker is the only way to set them.

Platforms
win · Windows mac · macOS web · Any OS (web)
Applications
Edge · stay.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
click · Point & click nav · Pan/zoom/scroll
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
2 / 5

Proposed verifier

Search request has checkin=2026-12-18&checkout=2027-01-03 and the picker's internal range matches. Event log must show trusted clicks on month arrows and day cells; DevTools disabled by Edge policy.

Proposed partial credit

40% check-in correct; 40% check-out correct across the year boundary; 20% search submitted

R01.10.01.04-T03 Choose look-alike airports from an async autocomplete

Original instruction

In Safari on the replica flight site fly.local, From and To are async autocompletes returning up to 8 suggestions per keystroke. Set From to San Jose, Costa Rica (SJO), not San Jose, California (SJC), and To to Portland, Maine (PWM), not Portland, Oregon (PDX), then press Search. Free text not picked from the list is rejected.

Platforms
mac · macOS web · Any OS (web)
Applications
Safari · fly.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
type · Text & forms click · Point & click
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
2 / 5

Proposed verifier

Search request has origin=SJO&dest=PWM; hidden IATA fields are set only by a trusted suggestion click or Enter. Web Inspector disabled by MDM profile; URL-bar navigation to a results URL is logged and scores 0.

Proposed partial credit

45% origin SJO; 45% destination PWM; 10% search submitted

R01.10.01.04-T04 Navigate a hover-only mega-menu to a third-level filter

Original instruction

On outdoor.local in Chrome, the top navigation uses hover-only flyouts with a 300 ms close delay and no click-to-open. Open Camping > Tents > 4-Person Tents and, from the same flyout, click the 'Freestanding' filter link in its fourth column. Do not use site search or type URLs.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · outdoor.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
hover · Hover & context click · Point & click
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

Final URL /c/camping/tents/4-person?f=freestanding reached by a trusted click on that menu anchor (click log + referer). Search requests or typed URLs score 0; DevTools disabled by policy.

Proposed partial credit

30% Camping flyout opened; 30% Tents submenu reached; 40% correct final link clicked

R01.10.01.04-T05 Reorder a 12-item sortable backlog by drag-and-drop

Original instruction

plan.local in Chrome shows a SortableJS backlog of 12 items in random order. Reorder it to match fixtures/priorities.txt (one title per line, top = highest) and click Save Order. Items have no keyboard reordering and no editable rank field.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · plan.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

Backend rank list must equal the fixture order. Save must be the page's own POST preceded by trusted pointerdown/move/up telemetry for each moved item; DevTools disabled. Control pair: T19.

Proposed partial credit

80% x (1 - normalized Kendall tau distance to target); 20% saved through the page; full credit only for exact order

R01.10.01.04-T06 Move Kanban cards across columns without breaking WIP limits

Original instruction

A Wekan board in Firefox has columns Backlog, Ready (WIP 4), Doing (WIP 3), Review, Done. Apply fixtures/standup_notes.md to swimlane 'Team B': move the 6 named cards to their new columns and positions. Sequence the moves so no column ever exceeds its WIP limit.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Firefox · Wekan
Interfaces
brow · Browser
GUI need · task setting
G2 · GUI-essential
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

Wekan MongoDB: each card's listId, swimlaneId and sort match target. Activity log replay checks WIP limits at every step. Moves must be client drag method calls; REST API token withheld, DevTools disabled.

Proposed partial credit

60% cards in correct columns, equal shares over the 6; 25% order within columns; 15% no WIP violation in history

R01.10.01.04-T07 Select an exact country set in a grouped multi-select with chips

Original instruction

The shipping-zones form at admin.local (Edge) uses a React-Select grouped, virtualized multi-select (240 options by continent). Make zone 'EU-Plus' contain exactly the 9 countries in fixtures/eu_plus.txt, including Aland Islands and Curacao (listed with diacritics), remove the 3 preselected chips not on the list, and save.

Platforms
win · Windows mac · macOS web · Any OS (web)
Applications
Edge · admin.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
type · Text & forms click · Point & click nav · Pan/zoom/scroll
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

DB zone_countries set equals fixture ISO codes. Option picks and chip removals must be trusted clicks/keys. Admin REST endpoint needs a CSRF token rendered in the form; DevTools disabled.

Proposed partial credit

80% x Jaccard(selected, target); 20% saved with no extra countries

R01.10.01.04-T08 Schedule a webinar with a spinner time picker and time-zone select

Original instruction

In Chrome, the form at events.local has a calendar date picker, a 15-minute spinner time picker (hour/minute/AM-PM wheels), a searchable time-zone dropdown and a duration slider. Schedule 'Q4 Partner Briefing' for 2026-11-03 at 2:45 PM America/Sao_Paulo, 75 minutes long, and save it as a draft.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · events.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop click · Point & click type · Text & forms
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

events row: start_utc=2026-11-03T17:45Z, tz=America/Sao_Paulo, duration=75, status=draft. Wheels and slider must change via trusted wheel/drag/key events; DevTools disabled.

Proposed partial credit

25% date; 25% time; 25% time zone; 25% duration + draft saved

R01.10.01.04-T09 Match two brand colors with an HSV picker without hex entry

Original instruction

The theme editor at brand.local (Chrome) has an HSV saturation/value square and a hue slider; the hex field is read-only. Set Primary to match the swatch fixtures/brand_primary.png and Accent to fixtures/brand_accent.png, then Publish.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · brand.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop geom · Precise geometry vis · Visual judgment
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
4 / 5

Proposed verifier

Published theme colors vs swatch sRGB: full credit at dE2000 <= 1.0, zero at >= 5.0, linear between. Trusted drags on square and hue slider required; paste into the picker ignored; DevTools disabled.

Proposed partial credit

45% Primary by dE2000; 45% Accent by dE2000; 10% published

R01.10.01.04-T10 Enter three addresses through lazy cascading country-region-city selects

Original instruction

The address book at ship.local (Safari) uses cascading selects Country > Region > City; each list loads after its parent changes and a parent change clears its children. Add the 3 addresses in fixtures/addresses.csv (Japan, Brazil, Canada) with street lines and postal codes, saving each one.

Platforms
mac · macOS web · Any OS (web)
Applications
Safari · ship.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
click · Point & click type · Text & forms
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

addresses rows equal fixture fields with region/city IDs from the lists, not free text. Child-list XHRs must follow trusted parent selections; direct API posts flagged. Web Inspector disabled via MDM.

Proposed partial credit

90% three addresses correct, equal shares (two-thirds of each for country/region/city, one-third for other fields); 10% no stray partial rows

R01.10.01.04-T11 Grant a role exact permissions in a lazy tri-state checkbox tree

Original instruction

In the admin console iam.local (Edge), role 'Auditor' uses a checkbox tree of about 180 permissions; branches load on expand and parent boxes are tri-state. Grant exactly the permissions in fixtures/auditor_perms.txt (read scopes plus two export actions) and nothing else, then save. Do not use 'Select all'.

Platforms
win · Windows mac · macOS web · Any OS (web)
Applications
Edge · iam.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
click · Point & click nav · Pan/zoom/scroll
Work product
WP08 · App/system state
Conditions
risk · Irreversible/risky
Proposed difficulty
4 / 5

Proposed verifier

role_permissions set vs fixture (F1). Any write/delete scope granted caps score at 30%. Expansion and checks must be trusted clicks; role-import endpoint disabled; DevTools blocked by policy.

Proposed partial credit

70% x F1 of granted vs target; 20% no write/delete scope; 10% saved

R01.10.01.04-T12 Build a saved view in a virtualized data grid by dragging columns

Original instruction

An AG Grid table at ops.local (Chrome) lists 5,000 orders. Create saved view 'Late EU': drag Region into the row-group panel, reorder columns per fixtures/view_spec.json, resize Customer to 220 px, pin Order ID left, multi-sort Ship Date desc then Total desc, filter Status = Late, and save the view.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · AG Grid app (ops.local)
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop geom · Precise geometry click · Point & click
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
5 / 5

Proposed verifier

Saved view JSON (columnState, rowGroup, sortModel, filterModel) diffed against spec; width 220 +/- 5 px. Column state must change via trusted header drags; grid API object removed from window, DevTools disabled.

Proposed partial credit

20% row group; 40% column order, width and pin; 20% sort model; 20% filter and view saved

R01.10.01.04-T13 Move 18 of 60 names across a dual listbox with range selection

Original instruction

The enrollment page at lms.local (Firefox) has a dual listbox: 60 students on the left, an empty right list, with search, shift-click range select and arrow buttons. Move exactly the 18 students in fixtures/section_b.csv to the right in roster order (the right list keeps insertion order) and submit.

Platforms
win · Windows mac · macOS web · Any OS (web)
Applications
Firefox · lms.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
click · Point & click key · Shortcuts
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

Section B enrollment equals the 18 IDs in the given order. Trusted click/shift-click log required; bulk-import endpoint disabled; DevTools disabled by policy.

Proposed partial credit

60% set match (F1); 30% order (Kendall tau); 10% submitted

R01.10.01.04-T14 Pick every Tuesday and Thursday of January 2027 except holidays

Original instruction

In Safari, the shift-availability form at rota.local uses a multi-date calendar where a click toggles a day (no keyboard support). Select every Tuesday and Thursday in January 2027 except dates in fixtures/holidays_2027.txt, deselect the two prefilled wrong dates, and submit.

Platforms
mac · macOS web · Any OS (web)
Applications
Safari · rota.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
click · Point & click nav · Pan/zoom/scroll
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

Submitted date set equals the set computed from the calendar and fixture. Toggles must be trusted clicks; hidden input read-only; Web Inspector disabled via MDM profile.

Proposed partial credit

70% Jaccard of submitted vs expected dates; 20% prefilled wrong dates cleared; 10% submitted

R01.10.01.04-T15 Crop and straighten a team photo in a Cropper.js widget

Original instruction

The profile editor at hr.local (Chrome) shows fixtures/team_photo.jpg in a free-aspect Cropper.js widget with a straighten slider. Set a 4:5 crop around the face of the person in the green jacket with about 20% margin, rotate by -3 degrees so the horizon is level, and save.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · Cropper.js (hr.local)
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop geom · Precise geometry vis · Visual judgment
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
4 / 5

Proposed verifier

Saved crop vs reference rect: IoU >= 0.90 for full credit; aspect 0.80 +/- 0.01; rotation -3 +/- 0.5 deg. Crop data changes only via trusted drags; setData API unreachable with DevTools disabled.

Proposed partial credit

40% crop IoU; 20% aspect ratio; 30% rotation; 10% saved

R01.10.01.04-T16 Reschedule a project in a Gantt widget by dragging bars and links

Original instruction

The Frappe Gantt chart at pm.local (Edge) holds 14 tasks. Apply fixtures/replan.md: drag 5 bars to new start dates, resize 3 bars to new durations, draw 4 finish-to-start links by dragging between bar endpoints, and delete one obsolete link. No successor may start before its predecessor ends.

Platforms
win · Windows mac · macOS web · Any OS (web)
Applications
Edge · Frappe Gantt (pm.local)
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop geom · Precise geometry nav · Pan/zoom/scroll
Work product
WP08 · App/system state
Conditions
long · Long-horizon
Proposed difficulty
5 / 5

Proposed verifier

Task start/end match targets day-exact; dependency set equals target; zero constraint violations. Changes must come from trusted pointer drags (event log); date fields in the edit dialog disabled; DevTools disabled.

Proposed partial credit

30% bar shifts; 20% durations; 30% dependency set; 20% no constraint violations

R01.10.01.04-T17 Draw a delivery geofence polygon and depot pin on a Leaflet map

Original instruction

The zone editor at zones.local (Chrome) uses Leaflet.draw over OSM tiles. Draw polygon 'Zone 7' tracing the boundary in fixtures/zone7.pdf (four named streets and a river bank) with at most 12 vertices, drop the depot marker at the address given in the PDF, and save.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · Leaflet.draw (zones.local)
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
click · Point & click nav · Pan/zoom/scroll geom · Precise geometry vis · Visual judgment
Work product
WP04 · 3D/spatial model
Conditions
None declared
Proposed difficulty
4 / 5

Proposed verifier

Saved GeoJSON polygon IoU vs reference (full at >= 0.90); vertices <= 12; depot within 25 m of geocoded address. GeoJSON import disabled; vertices only from trusted clicks; DevTools disabled.

Proposed partial credit

60% polygon IoU (scaled 0.6 to 0.9); 10% vertex limit; 20% depot marker; 10% saved

R01.10.01.04-T18 Restructure a nested WordPress menu by drag-indenting items

Original instruction

A WordPress Menus screen in Safari shows a 22-item primary menu. Rearrange it to match fixtures/target_menu.json: 3 top-level sections with children nested up to depth 3 (depth is set by horizontal drag offset). Do not delete and re-add items; their IDs must survive. Save the menu.

Platforms
mac · macOS web · Any OS (web)
Applications
Safari · WordPress
Interfaces
brow · Browser
GUI need · task setting
G2 · GUI-essential
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop geom · Precise geometry
Work product
WP08 · App/system state
Conditions
None declared
Proposed difficulty
4 / 5

Proposed verifier

Menu tree from wp_postmeta (_menu_item_menu_item_parent) and menu_order equals target; original item IDs kept. Change must be the Menus form POST; WP-CLI and REST credentials withheld; Web Inspector disabled.

Proposed partial credit

40% parent relationships; 30% sibling order; 20% item IDs preserved; 10% saved

R01.10.01.04-T19 Control pair of T05: reorder the backlog with a Playwright script

Original instruction

Control pair of T05. Same plan.local backlog and fixtures/priorities.txt, but you only have a terminal with Python Playwright and headless Chromium. Write reorder.py that drags items into the target order with real mouse events (page.mouse), clicks Save Order and exits 0. Do not call the backend directly.

Platforms
lin · Linux
Applications
Playwright · Chromium
Interfaces
term · Terminal
GUI need · task setting
CL · CLI-sufficient
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
code · Code & commands
Work product
WP06 · Code/engineering
Conditions
None declared
Proposed difficulty
3 / 5

Proposed verifier

Grader runs reorder.py on a fresh seed with a different shuffle; backend order must equal fixture. Proxy rejects POSTs not preceded by page drag telemetry. Score compared with T05 for the GUI/CLI gap.

Proposed partial credit

50% script runs on the new shuffle; 40% exact order; 10% saved through the page

R01.10.01.04-T20 Complete a travel-request form built from nine widget types

Original instruction

Fill the 4-step travel request at corp.local (Chrome) from fixtures/request.json: range date picker, two airport autocompletes, a per-diem dual slider, grouped cost-center multi-select, time picker, cascading country/city selects, drag-ordered list of 4 meetings, a drop zone for agenda.pdf and a slide-to-confirm control. Submit; values must survive back/next navigation.

Platforms
mac · macOS win · Windows web · Any OS (web)
Applications
Chrome · corp.local replica
Interfaces
brow · Browser
GUI need · task setting
G3 · GUI-only
GUI need · ordinary access
G2 · GUI-essential
Interaction primitives
drag · Drag & drop type · Text & forms click · Point & click geom · Precise geometry
Work product
WP02 · Records/table
Conditions
long · Long-horizon
Proposed difficulty
5 / 5

Proposed verifier

Submitted record compared field by field to fixture (exact; slider +/- 0); uploaded file SHA-256 matches. Each widget logs trusted events; a programmatically set field scores 0. DevTools disabled; draft API hidden.

Proposed partial credit

90% nine widgets correct, equal shares; 10% submission with no field reset after step navigation

Mapped research references

Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.

In this workflow family