R04.01.02.04 · TASK CLASS

Single-site lookup, aggregation & saved items

Find, count or aggregate facts inside one site or app, including the user's own account data (orders, posts, repos, reviews), and manage its saved items (followed locations, saved searches, watchlists).

20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.

Typical applications: shopping sites · GitLab · forums · admin consoles
Class GUI-need label: HY · Hybrid

Task proposals · 20

Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.

R04.01.02.04-T01 Total my non-canceled March orders on the shop replica

Original instruction

You are logged in to the OneStopShop replica storefront as test customer emma.lopez@shop.test. Find how many orders she placed in March 2023 that were not canceled and the sum of their grand totals. Write 'count,total' to answer.txt with the total to 2 decimals.

Platforms: web · Any OS (web)
Applications: OneStopShop (Magento storefront)
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Exact match against gold computed from the snapshot's sales_order rows for this customer (created_at in March 2023, state != canceled). Customer REST/GraphQL token endpoints are disabled and the agent has no shell, so order history is reachable only through the storefront's My Orders pages, which the access log must show.

Proposed partial credit

50% order count exact; 50% total within $0.01

R04.01.02.04-T02 Count a user's open bug issues across a GitLab group and find the top project

Original instruction

On the GitLab CE test instance, group 'a11yproject' has 7 projects. Count open, non-confidential issues authored by 'jdoe' that carry label 'bug' but not 'wontfix', across the whole group, and name the project holding the most of them. Write {"count": n, "top_project": "path"} to answer.json.

Platforms: web · Any OS (web)
Applications: GitLab
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Gold from the GitLab DB snapshot; count exact, project path exact. The agent holds no PAT, token creation is disabled by the admin, DevTools are disabled by browser policy and there is no shell, so data is reachable only through the rendered issue lists (grader flags /api/v4 requests lacking the frontend CSRF header).

Proposed partial credit

60% count exact; 40% top project correct

R04.01.02.04-T03 Find the most frequent author of busy f/DIY threads and their mean score

Original instruction

On the Postmill forum replica, look at f/DIY submissions from the 30 days before the snapshot date in task.txt that have at least 20 comments. Report the username who authored the most of them and the mean score of that user's qualifying submissions (2 decimals). Write 'user,mean' to answer.txt.

Platforms: web · Any OS (web)
Applications: Postmill
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: nav · Pan/zoom/scroll click · Point & click
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Gold computed from the Postmill DB snapshot; username exact, mean within 0.01. Postmill is server-rendered with no public API; the agent has no shell and DevTools are disabled, so listing pages must be paged in the UI (access log shows the f/DIY listing pages visited).

Proposed partial credit

30% qualifying set identified (grader checks visited submission pages); 40% correct user; 30% mean within 0.01

R04.01.02.04-T04 Top three California customers by completed Q1 order value in Magento admin

Original instruction

Magento 2 admin of the test store (login in creds.txt). For orders created 2023-01-01 to 2023-03-31 with status Complete and a California shipping address, find the 3 customers with the highest summed grand total. Write top3.csv with columns email,total (2 decimals), highest first. You may export grids to CSV and use LibreOffice Calc.

Platforms: web · Any OS (web) lin · Linux
Applications: Magento admin · LibreOffice Calc
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms multi · Cross-app
Work product: WP02 · Records/table
Conditions: multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

top3.csv compared to gold from the DB snapshot: emails and order exact, totals within $0.01. Grader also checks the admin action log for the order-grid filter or export, so answers guessed without reading the store score 0. Admin REST tokens are not issued.

Proposed partial credit

30% correct filtered order set (export present or grid filters logged); 40% correct 3 customers; 30% totals and order

R04.01.02.04-T05 Find the worst refund month of 2022 from Magento reports with stale statistics

Original instruction

In the Magento 2 test store admin, determine which month of 2022 had the highest refunded total, and that month's refunds as a percentage of invoiced total for the same month (1 decimal). Reports > Sales may show stale data. Write 'YYYY-MM,refunded,pct' to answer.txt.

Platforms: web · Any OS (web)
Applications: Magento admin
Interfaces: brow · Browser
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: hidden · Hidden/stale state
Proposed difficulty: 4 / 5

Proposed verifier

Gold from the DB snapshot (sales_creditmemo and sales_invoice by month). Report statistics were last refreshed in June 2022, so a correct answer requires refreshing statistics (logged in report_event) or computing from exported grids; month exact, refunded within $0.01, pct within 0.1.

Proposed partial credit

20% statistics refreshed or raw grids exported; 40% correct month; 20% refunded amount; 20% percentage

R04.01.02.04-T06 Count Q2 solved Support topics and those solved by staff on Discourse

Original instruction

On the Discourse test forum (logged in as member 'lena'), count topics in category 'Support' created between 2026-04-01 and 2026-06-30 that have an accepted solution, and list the IDs of those whose accepted answer was written by a staff member. Write {"solved": n, "staff_ids": [...]} to solved.json.

Platforms: web · Any OS (web)
Applications: Discourse
Interfaces: brow · Browser
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: type · Text & forms click · Point & click nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Gold from the Discourse DB snapshot (solved plugin custom fields, staff flags). Count exact; staff_ids scored by F1. Search filters in the UI and the forum's public .json routes are both allowed; no admin API key is provided.

Proposed partial credit

40% solved count exact; 60% staff_ids F1

R04.01.02.04-T07 Trace who last changed a SCSS variable and through which merge request

Original instruction

On the GitLab test instance, project 'design-system/primer-css': find who last changed the line defining $container-xl in src/support/variables/layout.scss on main, the 8-character commit SHA, and the merge request (!IID) that brought that commit into main. Write 'author,sha8,!iid' to answer.txt.

Platforms: web · Any OS (web)
Applications: GitLab
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll hover · Hover & context
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Exact match with gold from git blame on the snapshot repo plus the MR-commit link in the DB. No clone URL credentials or PAT are issued, token creation is disabled, the agent has no shell and DevTools are disabled, so blame and commit pages must be read in the UI.

Proposed partial credit

40% correct commit SHA; 30% author; 30% merge request IID

R04.01.02.04-T08 Count my starred GitLab projects and find the most-forked one

Original instruction

Logged in to the GitLab test instance as 'byteblaze', find how many projects you have starred and which starred project has the most forks. Write 'count,project_path' to answer.txt.

Platforms: web · Any OS (web)
Applications: GitLab
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Exact match with gold from users_star_projects and forks counts in the DB snapshot. No PAT is issued and token creation is disabled; with no shell and DevTools disabled the starred list must be read in the UI.

Proposed partial credit

50% star count; 50% most-forked project

R04.01.02.04-T09 Rank repeat WooCommerce customers by 2025 refund ratio

Original instruction

WooCommerce test store admin (creds.txt). Among customers with at least 5 orders in 2025, find the 3 with the highest refund ratio (refunded amount / gross sales). Write top_refund.csv with columns email,orders,ratio (3 decimals), highest first. WooCommerce Analytics downloads and a spreadsheet may be used.

Platforms: web · Any OS (web) mac · macOS
Applications: WooCommerce · Numbers
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click multi · Cross-app type · Text & forms
Work product: WP02 · Records/table
Conditions: multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Compared to gold from wc_order_stats/wc_order_product_lookup in the snapshot: 3 emails exact and ordered, orders count exact, ratio within 0.001. Analytics CSV download is allowed; no REST consumer key is issued, so the export must come from the admin UI (download logged).

Proposed partial credit

25% eligible customers identified; 25% refunds correctly attributed; 30% correct top 3; 20% ratios

R04.01.02.04-T10 List everyone who effectively holds a Keycloak client role

Original instruction

Keycloak 25 admin console, realm 'staff'. List every user who effectively holds client role 'billing-admin' of client 'erp': directly, through group or subgroup membership, or through a composite realm or client role. Note the console's 'Users in role' tab lists only direct holders. Write sorted usernames to billing_admins.txt.

Platforms: web · Any OS (web)
Applications: Keycloak
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Set compared to gold from the realm export (effective role mapping); score = F1. admin-cli direct grants are disabled, no service-account secret is issued, DevTools are disabled by browser policy and the agent has no shell; the Keycloak admin event log must show only console reads.

Proposed partial credit

30% direct holders; 40% group/subgroup holders; 30% composite-role holders (F1 per class)

R04.01.02.04-T11 Count 2025 deliveries with a brand item in an Android shopping app

Original instruction

On the Android emulator, the 'Kestrel Outdoor' replica shopping app is signed in to a test account. Count orders delivered in 2025 that contain at least one Kestrel-brand item, and their total paid. Put 'count; total' as the only text of the note titled 'answer' in the preinstalled Notes app.

Platforms: and · Android
Applications: Kestrel Outdoor (replica app) · Notes
Interfaces: mob · Mobile touch
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G3 · GUI-only
Interaction primitives: touch · Touch gestures nav · Pan/zoom/scroll multi · Cross-app
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader reads the Notes DB and compares with gold from the app backend DB. The backend API is certificate-pinned to the app, the agent has no adb shell or proxy, and order details render only in the app, so the answer must come from in-app screens.

Proposed partial credit

30% delivered 2025 orders found; 40% count exact; 30% total within 0.01

R04.01.02.04-T12 Median time-to-close of August bug issues via GitLab CSV export

Original instruction

Project 'infra/terraform-modules' on the GitLab test instance. Compute the median time from creation to close, in hours (1 decimal), of issues labeled 'bug' closed in August 2026, and the assignee who closed most of them. Use the issue list's CSV export (it arrives in the MailHog inbox at http://mail.test) and any spreadsheet. Write 'median,assignee' to answer.txt.

Platforms: web · Any OS (web) win · Windows
Applications: GitLab · MailHog · Excel
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click multi · Cross-app type · Text & forms
Work product: WP12 · Answer/decision
Conditions: multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Gold from the DB snapshot; median within 0.1 h, assignee exact. Grader checks that an export was requested (GitLab audit/export job log); no PAT is issued so the API route is closed, while UI and CSV/spreadsheet work are allowed.

Proposed partial credit

25% export obtained and filtered to August bugs; 50% median within 0.1 h; 25% assignee

R04.01.02.04-T13 Audit my 2022 orders for items whose price has since dropped

Original instruction

Logged in to the OneStopShop replica as test customer omar.k@shop.test. For every order placed in 2022, compare each item's paid unit price with its current product-page price. List items that are now cheaper in price_drops.csv with columns order_id,sku,paid,current,diff, sorted by diff descending.

Platforms: web · Any OS (web)
Applications: OneStopShop (Magento storefront)
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll multi · Cross-app
Work product: WP02 · Records/table
Conditions: long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Rows compared with gold joined from sales_order_item and current catalog prices (incl. special prices): F1 on (order_id, sku), values within $0.01. Customer token endpoints are disabled and the agent has no shell, so data must be read from the storefront pages (access log shows order and product views).

Proposed partial credit

30% all 2022 orders covered; 50% row F1 for price drops; 20% values and sort order

R04.01.02.04-T14 Count corporate reservations in a PMS reachable only by remote desktop

Original instruction

A legacy hotel PMS Windows client runs inside a noVNC remote desktop (browser tab 'PMS'). For property 'Harbor View', count November 2026 reservations with rate code CORP and status Guaranteed, and their total room-nights. Type 'count,nights' into answer.txt on the local desktop.

Platforms: win · Windows web · Any OS (web)
Applications: Hotel PMS (legacy client) · noVNC
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G3 · GUI-only
Interaction primitives: click · Point & click nav · Pan/zoom/scroll vis · Visual judgment
Work product: WP12 · Answer/decision
Conditions: multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Exact match with gold queried by the grader from the PMS database. The PMS exists only inside the remote session; clipboard and file transfer are disabled and no DB or API access is given, so values must be read from the remote screen and retyped.

Proposed partial credit

30% correct reservation filter applied (PMS UI log); 35% count; 35% room-nights

R04.01.02.04-T15 Find licensed users with no sign-in for 30 days in a dev tenant

Original instruction

Microsoft 365 admin center for the developer test tenant contoso-dev (global reader account). List users who hold an E5 license but have not signed in for 30 days or more as of the snapshot date, with their department. Write stale_e5.csv (upn,department) sorted by upn. The Active users export and Entra sign-in log views may be used.

Platforms: web · Any OS (web) win · Windows
Applications: Microsoft 365 admin center · Microsoft Entra admin center · Excel
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click multi · Cross-app nav · Pan/zoom/scroll
Work product: WP02 · Records/table
Conditions: multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

Set compared with gold from the tenant snapshot (license assignment and lastSignInDateTime): F1 on upn, department exact. Read-only role, so no state changes are possible; Graph PowerShell is not installed but UI exports plus spreadsheet joins are the intended route.

Proposed partial credit

30% E5 holders exported; 30% sign-in data joined; 40% final set F1 with departments

R04.01.02.04-T16 Measure unresolved bug backlog age per component with Jira JQL

Original instruction

Jira Software test site, project 'PAY'. For each component, count unresolved issues of type Bug with priority High or Highest and give the age in days of the oldest one as of the snapshot date. Write backlog.csv (component,count,oldest_days). Use JQL search and its CSV export.

Platforms: web · Any OS (web) mac · macOS
Applications: Jira · Numbers
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: type · Text & forms click · Point & click multi · Cross-app
Work product: WP02 · Records/table
Conditions: multiapp · Multi-app
Proposed difficulty: 3 / 5

Proposed verifier

Rows compared with gold from the site snapshot: per-component count exact and oldest_days within 1. No API token is issued (Atlassian REST closed), but JQL in the UI, CSV export and spreadsheet work are allowed; the export is logged in the audit log.

Proposed partial credit

30% correct JQL population; 40% per-component counts; 30% oldest ages

R04.01.02.04-T17 Find my busiest phpBB topic of 2025 and how many others replied

Original instruction

phpBB 3.3 test board, logged in as 'rivet'. Among topics in forum 'Hardware', find the one where rivet posted the most replies during 2025, then count how many distinct other users replied in that topic during 2025. Write 'topic_id,rivet_posts,other_users' to answer.txt.

Platforms: web · Any OS (web)
Applications: phpBB
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: type · Text & forms click · Point & click nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Exact match with gold from phpbb_posts in the snapshot. phpBB has no public read API, the agent has no shell or DB access and DevTools are disabled, so search results and topic pages must be read in the UI (search log shows the queries).

Proposed partial credit

40% topic identified; 25% rivet post count; 35% distinct other users

R04.01.02.04-T18 Control pair of T02: count the group's bug issues through the GitLab API

Original instruction

Control pair of T02. Same GitLab test instance and question: open, non-confidential 'bug' issues authored by 'jdoe' across group 'a11yproject', excluding 'wontfix', plus the project holding most. A read_api token is in ~/.config/gitlab_ro.txt; use curl or Python from the terminal. Write answer.json as in T02.

Platforms: lin · Linux
Applications: GitLab REST API · curl
Interfaces: term · Terminal api · API/SDK
GUI need · task setting: AP · API-sufficient
GUI need · ordinary access: AP · API-sufficient
Interaction primitives: code · Code & commands
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Same gold and scoring as T02 (count exact, project path exact). Grader confirms the answer came from /api/v4/groups/:id/issues calls in production_json.log; used only as the API-side control for T02.

Proposed partial credit

60% count exact; 40% top project correct

R04.01.02.04-T19 Build a monthly table of my forum posts, comments and score

Original instruction

On the Postmill forum replica as user 'quietfox', build activity.csv with one row per month from 2025-01 to 2025-12: month, submissions, comments, total score received on submissions created that month. Deleted items do not count. Only your own profile and forum pages are needed.

Platforms: web · Any OS (web)
Applications: Postmill
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: nav · Pan/zoom/scroll click · Point & click
Work product: WP02 · Records/table
Conditions: long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Each of 48 cells compared with gold from the Postmill DB snapshot (exact). Postmill has no read API; the agent has no shell and DevTools are disabled, so the profile's paged submission and comment lists must be read in the UI (access log shows the pages).

Proposed partial credit

30% submissions column; 30% comments column; 40% score column (pro-rata over months)

R04.01.02.04-T20 Find September orders whose discount exceeded the promo cap

Original instruction

Shopify development store admin (test store, staff account). input/promo_rules.pdf states a cap per discount code (percent and maximum amount). List every September 2026 order where the applied discount exceeds its code's cap, with the excess amount. Write over_cap.csv (order_name,code,discount,cap,excess). Order exports and a spreadsheet may be used.

Platforms: web · Any OS (web) win · Windows
Applications: Shopify admin · Excel
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: HY · Hybrid
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click multi · Cross-app type · Text & forms
Work product: WP02 · Records/table
Conditions: multiapp · Multi-app
Proposed difficulty: 5 / 5

Proposed verifier

Rows compared with gold computed from the store's order and discount data: F1 on order_name, excess within $0.01. Staff account has no app or API credentials; UI export of orders (logged in the store's activity log) plus spreadsheet joins is the intended route.

Proposed partial credit

20% caps read correctly from the PDF; 30% September orders with codes exported; 50% over-cap rows F1 with excess

Mapped research references

Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.

WebArena

Single-site lookup, aggregation & saved items

Task proposals · 20

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Mapped research references

In this workflow family