Product discovery, comparison & shopping lists
Find products that meet a stated need, compare offers across shops, keep wish lists and carts, and substitute out-of-stock items.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- shopping sites · grocery apps
- Class GUI-need label
HY· Hybrid
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R12.01.01.02-T01 Find the lowest delivered price for one headphone model across three shops
Original instruction
Across the ShopA, ShopB and ShopC clones, find the lowest total (item + shipping to ZIP 02139 + tax) for 'Sony WH-1000XM6, black, new condition'. Add one unit to the cart of the winning shop only and write the shop and total in answer.txt.
- Platforms
web· Any OS (web)- Applications
- ShopA clone · ShopB clone · ShopC clone
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickmulti· Cross-appvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
multiapp· Multi-app- Proposed difficulty
- 2 / 5
Proposed verifier
Frozen catalogs: answer.txt shop and total within $0.01 of ground truth (shipping only shown after entering ZIP at cart); winning cart has exactly 1 unit; other carts empty. Refurbished listings are traps.
Proposed partial credit
50% correct shop+total; 30% correct cart; 20% other carts empty.
R12.01.01.02-T02 Fill a grocery list across two stores to minimize total under a budget
Original instruction
Shopping list: 6x organic oat milk 1 L, 4x gluten-free pasta 500 g, 2x extra-virgin olive oil 750 ml, 1 kg Gala apples. Using the FreshCart and GreenGrocer clones, split items between stores to minimize the combined total including each store's delivery fee and minimum-order rules, budget $60. Fill both carts; stop before checkout.
- Platforms
web· Any OS (web)- Applications
- FreshCart clone · GreenGrocer clone
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickmulti· Cross-apptype· Text & forms- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Carts compared to the ILP optimum over frozen catalogs (delivery fees + minimum order thresholds); full credit if combined total equals optimum; partial if feasible but higher. No orders placed.
Proposed partial credit
40% all list items covered with exact quantities; 40% optimal total (half credit if within five percent); 20% both carts valid, no orders.
R12.01.01.02-T03 Substitute an out-of-stock ink cartridge with a listed compatible part
Original instruction
The user's printer is an 'Epsilon WF-3820'. Its OEM black cartridge is out of stock on the office-supply clone. Using the manufacturer's compatibility page on the brand-site clone, find a compatible in-stock black cartridge (OEM preferred over third-party) at either store and add it to cart.
- Platforms
web· Any OS (web)- Applications
- office supply clone · brand site clone
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickmulti· Cross-appnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 2 / 5
Proposed verifier
Cart SKU in the allowed compatibility set for WF-3820; OEM in-stock option exists at store 2 (full credit), third-party compatible = 50%. Incompatible or out-of-stock items = 0.
Proposed partial credit
50% compatible SKU; 30% OEM preference; 20% in stock and single line.
R12.01.01.02-T04 Build a comparison sheet of five air fryers across three shops
Original instruction
For the 5 air-fryer models in models.txt, record price, capacity (L), wattage and rating at ShopA, ShopB and ShopC clones in compare.xlsx (LibreOffice Calc), one row per model-shop pair, then mark the best value per model (lowest price per liter among listings rated >= 4.0).
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- LibreOffice Calc · shop clones
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-apptype· Text & formsvis· Visual judgment- Work product
WP02· Records/table- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
compare.xlsx: 15 rows vs frozen snapshot (price exact, capacity/watt exact, rating ±0.05); best-value flag correct per model. Some specs appear only in product images (spec card PNG).
Proposed partial credit
60% cell accuracy (proportional over 60 cells); 25% best-value flags; 15% sheet structure.
R12.01.01.02-T05 Fill a cart to qualify for free shipping with minimal overspend from favorites
Original instruction
On the grocery clone, the cart has $31.20 of items. Using only items from the user's 'Favorites' list, add items so the subtotal reaches the $50 free-shipping threshold with the smallest possible overspend. Stop before checkout and report the final subtotal.
- Platforms
web· Any OS (web)- Applications
- grocery store clone
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Final subtotal equals the subset-sum optimum ($50.37 seeded) and only favorites added; reported subtotal matches cart. Existing lines unchanged.
Proposed partial credit
40% threshold reached with favorites only; 40% optimal overspend; 20% report + original lines intact.
R12.01.01.02-T06 Compare return policies for opened electronics and choose the longest window
Original instruction
Find which of the three shop clones offers the longest return window for opened electronics (read each store's policy page, including exceptions for headphones), then add 'Bose QC Ultra' to that store's cart and save the policy excerpt you used to policy.txt.
- Platforms
web· Any OS (web)- Applications
- shop clones
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollmulti· Cross-appclick· Point & click- Work product
WP12· Answer/decision- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Cart in correct store (headphones exception changes winner from ShopA to ShopC); policy.txt contains the exception sentence (fuzzy match >= 0.9). Other carts empty.
Proposed partial credit
50% correct store accounting for exception; 30% cart; 20% excerpt saved.
R12.01.01.02-T07 Scale a recipe for 6 servings and cart all ingredients with substitutions
Original instruction
Recipe fixtures/ramen.pdf is for 2 servings. Scale to 6 and add all 11 ingredients to the grocery clone cart in pack sizes covering the scaled amounts with the least waste. If an ingredient is out of stock, use the substitute named in the recipe's notes. Stop before checkout.
- Platforms
web· Any OS (web)- Applications
- grocery store clone · PDF viewer
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentmulti· Cross-app- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Cart lines: 11 ingredients (2 via substitutes), quantities cover scaled amounts with minimal packs (checker per ingredient). Extra items = penalty.
Proposed partial credit
45% ingredients present (per item); 35% pack-size minimality; 20% correct substitutions.
R12.01.01.02-T08 Reorder a past order minus discontinued items and report skipped ones
Original instruction
On the grocery app clone (Android emulator), reorder everything from order #5521 except items now out of stock or discontinued. For each skipped item, add the closest same-brand alternative if one exists, else skip. Write skipped items and replacements into the app's notes field of the cart.
- Platforms
and· Android- Applications
- grocery app clone
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturesclick· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Cart lines equal expected set (in-stock originals + same-brand replacements); cart note lists the 3 skipped items and 1 replacement (string match). Bulk 'reorder' button is disabled in the seeded app version.
Proposed partial credit
50% cart lines correct (per line); 25% replacements; 25% note accurate.
R12.01.01.02-T09 Assemble a back-to-school list from a teacher PDF across two shops under budget
Original instruction
fixtures/supply_list.pdf has 14 items with brand and count requirements. Using the office-supply clone and the general-store clone, fill carts so every item is covered, brand constraints met where marked 'required', and the combined total including shipping is minimal and <= $85.
- Platforms
web· Any OS (web)- Applications
- office supply clone · general store clone
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appclick· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
multiapp· Multi-applong· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Constraint checker on both carts (14 items, counts, required brands); total compared to ILP optimum over frozen catalogs; partial credit if feasible within 5%.
Proposed partial credit
40% coverage (per item); 20% brand constraints; 40% optimality.
R12.01.01.02-T10 Find same-day pickup availability for a blender within five miles
Original instruction
Which of the ShopA/ShopB/ShopC clones has the 'Vitamix E310' in stock for same-day pickup at a store within 5 miles of ZIP 94110? Store pickers show a map with pins. Reserve it for pickup at that store (no payment) and record the store number in answer.txt.
- Platforms
web· Any OS (web)- Applications
- shop clones
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt store id equals ground truth; pickup reservation row exists at that store with status=held; distance computed from seeded coordinates. Store distances render only on the map popup. The shop clones issue no API token and accept reservations only from browser sessions (CSRF + session id), so scripted holds are rejected.
Proposed partial credit
50% correct store; 40% reservation held; 10% no reservations elsewhere.
R12.01.01.02-T11 Price-match a TV by gathering competitor evidence screenshots
Original instruction
ShopA offers a price match if a competitor lists the identical model (same model number suffix) cheaper. For 'LG C4 55" OLED55C4PUA', check ShopB and ShopC, capture a screenshot of a qualifying listing to evidence.png, and submit ShopA's price-match form with the competitor URL and price, stopping before final submit.
- Platforms
mac· macOSwin· Windowsweb· Any OS (web)- Applications
- Chrome · shop clones
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appvis· Visual judgmenttype· Text & forms- Work product
WP08· App/system state- Conditions
multiapp· Multi-apprisk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Form draft: competitor URL equals the qualifying listing (ShopC's OLED55C4PUA; ShopB's lower price is model ...PSA, non-qualifying), price exact; evidence.png OCR contains model number and price. Draft not submitted.
Proposed partial credit
40% qualifying listing chosen; 30% form fields; 20% screenshot evidence; 10% not submitted.
R12.01.01.02-T12 Plan a weekly grocery order from a meal plan and pantry inventory
Original instruction
meal_plan.xlsx lists 7 dinners with ingredients; pantry.csv lists what is already at home. Compute what to buy and fill the grocery clone cart with exact needed quantities in available pack sizes, choosing store brand when cheaper. Stop before checkout and save shopping_list.csv of what you carted.
- Platforms
mac· macOSwin· Windows- Applications
- Excel · grocery store clone
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appcode· Code & commandsclick· Point & click- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Expected buy-list from plan minus pantry (unit conversions); cart lines must cover it with minimal packs; store-brand choice where cheaper; shopping_list.csv equals cart. Per-ingredient scoring.
Proposed partial credit
40% correct buy-list; 40% cart coverage/minimal packs; 20% csv matches cart.
R12.01.01.02-T13 Compare unit prices of detergent sizes across shops including multi-buy promos
Original instruction
Find the lowest price per wash for 'Tide Free & Gentle' liquid across ShopA, ShopB and ShopC clones, considering all sizes and multi-buy deals ('2 for $25', 'buy 3 get 1'). Buy enough to cover at least 200 washes at the lowest total cost; add to that shop's cart only.
- Platforms
web· Any OS (web)- Applications
- shop clones
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appvis· Visual judgmentclick· Point & click- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Cart: wash count >= 200 and total equals optimum over frozen offers incl. promos (promos apply only in cart); other carts empty. Wash counts appear only on package images.
Proposed partial credit
30% wash counts read correctly (via cart coverage); 50% optimal cart; 20% single shop.
R12.01.01.02-T14 Add a complementary accessory compatible with the camera already in the cart
Original instruction
The electronics clone cart contains one camera body. Add the cheapest spare battery and a compatible 128 GB card meeting the camera's video-recording speed requirement listed in its manual PDF (linked on the product page). Do not change the existing cart line.
- Platforms
web· Any OS (web)- Applications
- electronics store clone
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Cart: original line unchanged; battery SKU in compat set and cheapest; card capacity 128 GB with speed class >= manual requirement (V60) and cheapest such.
Proposed partial credit
40% correct battery; 40% correct card; 20% original line untouched.
R12.01.01.02-T15 Fulfil a party shopping list on a mobile grocery app with substitutions by rule
Original instruction
In the grocery app clone on iOS Simulator, add the 12 items in the Notes app note 'Party' to the cart. Rule: if an item is out of stock, substitute the same category item with the closest price; if none within ±20%, skip and add it to the note's 'Skipped' section. Stop before checkout.
- Platforms
ios· iOS/iPadOS- Applications
- grocery app clone (iOS) · Notes
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturesmulti· Cross-apptype· Text & forms- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Cart lines equal expected (10 direct, 1 substitute, 1 skipped); Notes DB: 'Skipped' section lists the skipped item. Quantities exact. The agent drives only the Simulator window: no host shell (so simctl and the app/Notes data containers are not reachable) and no backend API token, so cart and note edits must be made in the apps.
Proposed partial credit
50% direct items; 20% substitution; 20% skipped noted; 10% no extras.
R12.01.01.02-T16 Compare a basket of four books across stores and buy the whole basket at one shop
Original instruction
For the 4 ISBNs in books.txt, find which single store (ShopA/B/C clones) gives the lowest total for the whole basket including shipping (some stores have free shipping over $35, one has a members' price after free sign-up — do not sign up). Fill that store's cart only.
- Platforms
web· Any OS (web)- Applications
- shop clones
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appclick· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Winning store cart has the 4 ISBNs exactly; totals computed without member prices; no new account created (users table unchanged). Other carts empty.
Proposed partial credit
40% correct store; 40% cart lines; 20% no sign-up, other carts empty.
R12.01.01.02-T17 Restock household supplies from a photo of the pantry shelf
Original instruction
fixtures/shelf.jpg shows a pantry with partly used items; restock_rules.txt gives target counts. Count what is visible, compute shortfalls, and add exactly the shortfall quantities to the grocery clone cart using the same brands as in the photo when available. Stop before checkout.
- Platforms
mac· macOSwin· Windowsweb· Any OS (web)- Applications
- grocery store clone · Photos
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentmulti· Cross-appclick· Point & click- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Cart quantities equal (target - visible count) for 9 products (ground-truth counts annotated); brand match where in stock. Per-product scoring. The store issues no API token and accepts cart writes only from the browser session (CSRF + session id), so scripted adds are rejected.
Proposed partial credit
40% counts correct (via quantities); 40% brands; 20% no extra items.
R12.01.01.02-T18 Compare mobile plans across carrier clones and save a ranked table
Original instruction
Usage: 18 GB data/month, 2 lines, international calls to Mexico 60 min/month. Compare plans on three carrier-site clones including fees, autopay discounts and promo expiry (total cost over 24 months). Save ranking.csv with the top 3 plans and their 24-month totals, and add the best plan to its carrier's cart without checking out.
- Platforms
web· Any OS (web)- Applications
- carrier site clones
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appnav· Pan/zoom/scrollcode· Code & commands- Work product
WP12· Answer/decision- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
ranking.csv top 3 plan IDs in correct order; totals within $1 of reference; best plan in cart with 2 lines; no checkout. Promo terms are behind 'details' expanders.
Proposed partial credit
30% plan totals; 30% ranking order; 30% cart; 10% no checkout.
R12.01.01.02-T19 Coordinate a family shopping list across chat and two store apps
Original instruction
On the Android emulator, three family members posted wanted items in the 'Family' chat (messaging clone) over the past week, including two retractions ('never mind the batteries'). Compile the final list, then fill carts in the grocery app and the pharmacy app clones (items sorted by which store sells them). Stop before checkout and post the final list back to the chat.
- Platforms
and· Android- Applications
- messaging clone · grocery app clone · pharmacy app clone
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturesmulti· Cross-appnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
multiapp· Multi-apppersonal· Personal contextlong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Expected final list (14 items after retractions); carts contain each item in the store that sells it with quantities; chat has a final-list message equal to the set. Per-item scoring. The agent gets screen input only (no adb shell, no backend API tokens), so carts and the chat post must be made in the apps; the harness input log confirms UI taps.
Proposed partial credit
30% list compilation incl. retractions; 50% carts (per item); 20% chat post.
R12.01.01.02-T20 Optimize a multi-shop order with coupons, free-shipping thresholds and stock limits
Original instruction
Buy the 9 items in party_order.csv across ShopA, ShopB and ShopC clones at minimum total. Each shop has its own coupon (on the Deals page), free-shipping threshold, and per-customer quantity limits on some items. Fill the carts, apply coupons, record your plan and expected total in plan.md, and stop before checkout.
- Platforms
web· Any OS (web)- Applications
- shop clones
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appclick· Point & clickcode· Code & commands- Work product
WP08· App/system state- Conditions
multiapp· Multi-applong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Combined cart total (computed by each shop's backend incl. coupons/shipping) compared with ILP optimum; quantity limits respected; plan.md total equals actual. Full credit at optimum, 50% within 3%.
Proposed partial credit
30% all items covered within limits; 20% coupons applied; 40% optimality; 10% plan consistent.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.