Lodging under constraints
Book stays meeting price, location, policy and date constraints.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Booking · Airbnb · hotel sites
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R12.02.01.02-T01 Book a stay with a crib within a ten-minute walk of the station
Original instruction
On the Staynb clone, book 3 nights 2026-12-18 to 12-21 for 2 adults and 1 infant in Kyoto, with a crib available, at most a 10-minute walk to Kyoto Station on the listing's map, total <= $700 including fees. Choose the cheapest qualifying listing and pay with the sandbox card.
- Platforms
web· Any OS (web)- Applications
- Staynb clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Reservation: dates, guests incl. infant, listing in feasible set (crib amenity, walking distance computed from seeded coordinates <= 800 m, total incl. fees <= 700) and cheapest of that set. Walking distance shown only on map overlay. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
20% dates/guests; 30% crib + distance; 30% cheapest feasible; 20% total within cap.
R12.02.01.02-T02 Reserve a hotel with free cancellation and breakfast at the best rating
Original instruction
On the hotel OTA clone, reserve a double room in Lisbon 2026-11-05 to 11-09 for 2 adults with free cancellation until at least 48 h before arrival and breakfast included, guest rating >= 8.5, and the lowest total. Pay at property (no card charge); enter guest details from fixtures/guest.json.
- Platforms
web· Any OS (web)- Applications
- hotel OTA clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Reservation row: rate plan has refundable_until >= arrival-48h, meal=breakfast, property rating >= 8.5, lowest total among such; payment=pay_at_property; guest fields exact. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
40% rate-plan constraints; 30% lowest; 15% payment type; 15% guest fields.
R12.02.01.02-T03 Extend an existing booking by one night if the same room is free
Original instruction
On the Staynb clone, the user has booking STY-8812 (Dec 3-6). Extend it to Dec 7 if the same listing is available that night; otherwise report 'unavailable' and do not create any new booking. Pay the extension with the sandbox card if extended.
- Platforms
web· Any OS (web)- Applications
- Staynb clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 2 / 5
Proposed verifier
Seeded variant A (available): booking end date 12-07, extra night charged once. Variant B (unavailable): no changes, answer 'unavailable'. Both variants scored; no duplicate bookings. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
60% correct branch outcome; 25% payment/no payment; 15% no duplicates.
R12.02.01.02-T04 Choose an apartment from a map within a polygon neighborhood boundary
Original instruction
On the vacation-rental clone, find the cheapest entire apartment in Lisbon's Alfama neighborhood (use the neighborhood outline overlay on the map; listings outside it do not count) for 3 adults Nov 5-9, with a washer and free cancellation. Book it with the sandbox card.
- Platforms
web· Any OS (web)- Applications
- vacation rental clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Booked listing id inside the seeded Alfama polygon (point-in-polygon), amenities washer, cancellation free, cheapest among such. Listing text claims 'Alfama' for two out-of-polygon decoys. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
40% inside polygon; 20% amenities/cancellation; 30% cheapest; 10% booked.
R12.02.01.02-T05 Fall back to the closest four-star hotel when the preferred one is sold out
Original instruction
Book the 'Harborview Grand' in Halifax for 2026-12-10 to 12-12, 1 king room. If it is sold out (check), book the closest 4-star hotel within 1 km on the same dates with a refundable rate instead. Pay with the sandbox card and report which branch you took.
- Platforms
web· Any OS (web)- Applications
- hotel OTA clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded sold-out: reservation at fallback property = nearest 4-star within 1 km (distances via seeded coords) with refundable rate; answer mentions sold-out branch. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
30% sold-out detected; 40% correct fallback; 20% refundable rate; 10% answer.
R12.02.01.02-T06 Book an accessible room with roll-in shower and verify via room photos
Original instruction
On the hotel OTA clone, book 2 nights in Chicago (2026-11-14 to 11-16) at the cheapest hotel within 2 km of McCormick Place offering an accessible room with a roll-in shower. Several listings claim 'accessible' but photos show tub-only bathrooms; verify via room photos and book the right room type.
- Platforms
web· Any OS (web)- Applications
- hotel OTA clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentclick· Point & clicknav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Reservation room_type has roll_in_shower=true in ground truth (photo-verified annotation), property within 2 km, cheapest such. Decoy room types with tubs = 0. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
50% correct accessible room; 30% cheapest within distance; 20% dates and booking.
R12.02.01.02-T07 Book lodging for a group of eight with bed-type constraints
Original instruction
On the Staynb clone, book a house in Asheville for 8 adults, 2026-11-26 to 11-29, with at least 4 bedrooms, no more than 2 people per bed, no sofa beds, a hot tub, and total <= $2,400. Bed configurations are listed in the 'Where you'll sleep' carousel. Book the cheapest valid listing.
- Platforms
web· Any OS (web)- Applications
- Staynb clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Listing satisfies bed-count predicate from seeded bed configs (carousel-only), hot tub amenity, total incl. fees <= 2400, cheapest valid; guest count 8. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
40% bed constraints; 20% amenities; 30% cheapest; 10% guests/dates.
R12.02.01.02-T08 Add loyalty number and special requests to an existing hotel reservation
Original instruction
On the hotel chain clone, open reservation HX-55102, add the user's loyalty number from fixtures/loyalty_card.png, request a high floor and late check-out at 14:00, and change the bed preference from two queens to one king without changing the rate.
- Platforms
web· Any OS (web)- Applications
- hotel chain clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmenttype· Text & formsclick· Point & click- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Reservation: loyalty_id exact (read from card image), requests contain high_floor and late_checkout=14:00, bed=king, rate code unchanged. The chain clone issues no API token and accepts reservation edits only from the browser session (CSRF + session id), so scripted updates are rejected.
Proposed partial credit
30% loyalty id; 30% requests; 30% bed change; 10% rate unchanged.
R12.02.01.02-T09 Book a hotel under a corporate per-diem cap with city tax included
Original instruction
Book 3 nights in Paris 2026-11-18 to 11-21 on the corporate booking clone for one traveler. The policy cap is EUR 210 per night including city tax and VAT (taxes appear only on the final price breakdown). Choose the highest-rated hotel within 3 km of La Défense meeting the cap, with a refundable rate, and complete the booking.
- Platforms
web· Any OS (web)- Applications
- corporate booking clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Reservation property = highest rating among hotels with nightly all-in <= 210 (computed incl. tax rules), within 3 km, refundable rate; a decoy is 205 + 6.50 tax. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
40% cap respected incl. taxes; 30% best rating; 20% distance + refundable; 10% booked.
R12.02.01.02-T10 Book on a mobile app with a date-range calendar and guest picker
Original instruction
In the Booking-style app clone on the iOS Simulator, book a room in Barcelona from Friday 2026-11-27 to Monday 11-30 for 2 adults and 2 children ages 4 and 9, with a family room or two connecting rooms, pool, and a rating >= 8.0. Pick the cheapest, pay with the sandbox card.
- Platforms
ios· iOS/iPadOS- Applications
- Booking-style app clone (iOS)
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturesclick· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Backend reservation: dates, 2 adults + 2 children with ages 4 and 9, room configuration valid, pool amenity, rating >= 8.0, cheapest; scroll-wheel age pickers must be set. The agent drives only the Simulator window: it has no shell on the host (simctl and the app container are not reachable) and no API token, so the booking must be made in the app.
Proposed partial credit
30% dates/guests incl. ages; 30% room configuration; 30% cheapest; 10% booked.
R12.02.01.02-T11 Book a lodge where availability is split across two room types
Original instruction
On the lodge booking clone, the user wants 5 nights 2026-12-27 to 2027-01-01 at 'Pine Ridge Lodge'. No single room is free all 5 nights. Find the split stay with the fewest room changes and lowest total, book both segments in one reservation if the site supports it, and pay with the sandbox card.
- Platforms
web· Any OS (web)- Applications
- lodge booking clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentclick· Point & clicknav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Reservation segments cover all 5 nights with exactly 1 room change (minimum) and minimal total among 1-change plans (from seeded availability grid shown as a calendar heatmap). The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
30% full coverage; 30% minimal changes; 30% lowest total; 10% single reservation.
R12.02.01.02-T12 Pick the hotel that minimizes transit time to two conference venues
Original instruction
The user attends sessions at two venues in Berlin (addresses in agenda.pdf), alternating days. On the hotel OTA clone, choose a hotel for 2026-11-09 to 11-13 under EUR 180/night that minimizes the sum of public-transit times to both venues (use the map clone's transit planner). Book a refundable rate.
- Platforms
mac· macOSwin· Windowsweb· Any OS (web)- Applications
- hotel OTA clone · map clone · PDF viewer
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appnav· Pan/zoom/scrollvis· Visual judgment- Work product
WP08· App/system state- Conditions
multiapp· Multi-apprisk· Irreversible/risky- Proposed difficulty
- 5 / 5
Proposed verifier
Booked property = argmin over price-feasible hotels of transit time sums computed by the same router on the seeded snapshot; ties within 2 min accepted; refundable. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
50% optimal hotel (half credit if top-3); 30% price cap; 20% refundable rate.
R12.02.01.02-T13 Modify a reservation to add a guest while keeping the non-refundable rate
Original instruction
Booking STY-4410 on the Staynb clone is non-refundable for 2 guests. Add a third guest via the 'Change reservation' flow, accept the extra-guest fee, and message the host through the in-app inbox asking for an extra set of towels. Do not cancel or rebook.
- Platforms
web· Any OS (web)- Applications
- Staynb clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Reservation guests=3, extra fee line added, rate plan unchanged, no cancellations; host message thread contains towel request. The clone issues no API token and accepts reservation changes and host messages only from the browser session (CSRF + session id), so scripted calls are rejected.
Proposed partial credit
50% guest added via change; 20% fee accepted; 15% no cancellation; 15% host message.
R12.02.01.02-T14 Book a pet-friendly stay verifying weight limits in house rules
Original instruction
On the vacation-rental clone, book 4 nights in Portland 2026-12-02 to 12-06 for 2 adults with a 32 kg dog. Many listings say 'pets allowed' but the house rules (expandable) set weight limits or fees. Choose the cheapest listing that allows a 32 kg dog including pet fees, fenced yard required.
- Platforms
web· Any OS (web)- Applications
- vacation rental clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Listing satisfies pet weight limit >= 32 kg (house rules text), fenced yard amenity, cheapest total incl. pet fee; pet count 1 in reservation. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
40% pet rule; 20% yard; 30% cheapest incl. fees; 10% reservation details.
R12.02.01.02-T15 Compare a hotel's direct rate with an OTA rate and book the cheaper equivalent
Original instruction
For the 'Marlow Inn' in Boston 2026-11-20 to 11-22, compare the direct-site clone rate (member rate after free sign-in with the test account) and the OTA clone rate for the same room type and cancellation terms. Book the cheaper equivalent offer and save the comparison as compare.txt with both totals.
- Platforms
web· Any OS (web)- Applications
- hotel direct site clone · hotel OTA clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appvis· Visual judgmentclick· Point & click- Work product
WP08· App/system state- Conditions
multiapp· Multi-apprisk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
compare.txt totals equal both backends' quotes for matching room+terms; booking on the cheaper site; non-equivalent cheaper OTA rate (non-refundable) must be ignored. Neither clone issues an API token; quotes and bookings are served only to browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
40% equivalent offers identified; 30% totals correct; 30% booked on cheaper site.
R12.02.01.02-T16 Book a ryokan on a Japanese-language mobile site with meal plan options
Original instruction
On the Japanese-language ryokan booking clone (Android emulator, Chrome), book 1 night in Hakone on 2026-11-21 for 2 adults with dinner and breakfast (1泊2食), a room with a private open-air bath (露天風呂付き客室), check-in 15:00. Cheapest qualifying. Stop at the payment step.
- Platforms
and· Android- Applications
- ryokan booking clone · Chrome
- Interfaces
mob· Mobile touchbrow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
touch· Touch gesturesvis· Visual judgmentclick· Point & click- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Draft reservation: plan meal=2 meals, room feature private_rotenburo=true, check-in 15:00, cheapest feasible; status=awaiting_payment. The agent gets screen input only (no adb shell), and the site issues no API token and accepts reservations only from the Chrome session (CSRF + session id).
Proposed partial credit
30% meal plan; 30% room feature; 30% cheapest; 10% stop at payment.
R12.02.01.02-T17 Plan lodging for a two-city trip within a total budget across sites
Original instruction
Book lodging for Rome (Nov 10-13) and Florence (Nov 13-16) for 2 adults, total budget EUR 1,100 across both, each within 1 km of the main train station (Termini / SMN), rating >= 8.0, refundable. Use either the hotel OTA clone or the Staynb clone per city to minimize the combined total; book both with the sandbox card.
- Platforms
web· Any OS (web)- Applications
- hotel OTA clone · Staynb clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appnav· Pan/zoom/scrollvis· Visual judgment- Work product
WP08· App/system state- Conditions
multiapp· Multi-apprisk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Two reservations; each satisfies distance/rating/refundable; combined total <= 1100 and equals min over the feasible pairs from both sites. Neither clone issues an API token; bookings are accepted only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
50% per-city constraints (half each); 40% combined optimum; 10% both booked.
R12.02.01.02-T18 Cancel a hotel booking within the free window and report the refund
Original instruction
On the hotel OTA clone, cancel reservation BK-77310 (Nov 3) whose free-cancellation deadline is tomorrow 18:00 local hotel time (the hotel is in a different time zone). Confirm it is still within the window using the property time zone, cancel, and report the refund amount and cancellation number.
- Platforms
web· Any OS (web)- Applications
- hotel OTA clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Reservation status=cancelled, refund equals full amount; answer contains refund and cancellation number equal to DB. Other reservations unchanged. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
50% correct reservation cancelled; 30% refund/number reported; 20% others untouched.
R12.02.01.02-T19 Book an extended-stay apartment with monthly discount and a lease-style form
Original instruction
On the vacation-rental clone, book a furnished 1-bedroom in Austin for 2026-12-01 to 2027-01-15 with a dedicated workspace and fast Wi-Fi (>= 100 Mbps per the listing's speed-test screenshot), applying monthly discounts. Complete the host's extended-stay questionnaire from fixtures/tenant.json and request to book (host approval required).
- Platforms
web· Any OS (web)- Applications
- vacation rental clone
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmenttype· Text & formsclick· Point & click- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Booking request on a listing with speed-test image showing >= 100 Mbps (annotated), workspace amenity, cheapest after monthly discount; questionnaire fields equal tenant.json; status=pending_host. The sandbox site issues no API token and accepts reservation writes only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
30% Wi-Fi verified from image; 30% cheapest after discount; 30% questionnaire; 10% request status.
R12.02.01.02-T20 Assemble lodging for a ten-person offsite across three properties within constraints
Original instruction
Book lodging for a 10-person team offsite in Denver 2027-02-08 to 02-11: everyone within a 5-minute walk of the 'Union Station' venue on the map, at most 3 properties, each person in their own bedroom (rooming list in team.csv with 2 accessible-room needs), total <= $6,000. Use the hotel OTA and Staynb clones; book everything with the sandbox card.
- Platforms
mac· macOSwin· Windowsweb· Any OS (web)- Applications
- hotel OTA clone · Staynb clone · LibreOffice Calc
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appnav· Pan/zoom/scrollvis· Visual judgment- Work product
WP08· App/system state- Conditions
multiapp· Multi-applong· Long-horizonrisk· Irreversible/risky- Proposed difficulty
- 5 / 5
Proposed verifier
Reservations cover 10 bedrooms in <= 3 properties within 400 m walking of venue (seeded coordinates), 2 accessible rooms assigned to the right people (guest names), total <= 6000. Feasibility checker; bonus for minimal total. Neither clone issues an API token; bookings are accepted only from browser sessions (CSRF + session id), so scripted requests are rejected.
Proposed partial credit
30% all 10 housed in own rooms; 20% distance; 20% accessible assignment; 30% budget cap met, with a third of this for minimal total.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.