UI test automation
Write and repair Playwright/Appium/XCUITest suites against a live app.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Playwright · Appium · XCUITest
- Class GUI-need label
HY· Hybrid
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R02.07.01.02-T01 Write Playwright tests that catch checkout regressions
Original instruction
For the sandbox shop at http://localhost:3000, write Playwright (TypeScript) tests in tests/checkout.spec.ts covering login, add to cart, promo code, shipping choice and order confirmation, using role/label locators. Inspect the running app to choose assertions.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright · Chrome
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
code· Code & commandsclick· Point & clickvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs the suite on the golden build (must pass 3/3 runs) and on 8 hidden mutant builds (wrong totals, disabled button, missing toast…): score = mutants killed. Static check: no waitForTimeout, no CSS nth-child selectors.
Proposed partial credit
20% passes on golden; 60% mutant kill rate; 20% locator/wait hygiene
R02.07.01.02-T02 Repair a Playwright suite broken by a UI redesign
Original instruction
After the v2 redesign of the sandbox admin app, 12 of 30 Playwright tests in e2e/ fail. Update locators and flows to the new UI (inspect it in the browser), without weakening assertions or adding sleeps. Report any test that fails because of a real app bug in output/app_bugs.md instead of changing its assertion.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright · Chrome
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
code· Code & commandsclick· Point & clickvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader: 29 tests pass on v2 golden; the 1 seeded real app bug must be reported (not 'fixed' by relaxing the test). Assertion count per test must not decrease (AST diff); no waitForTimeout. Suite still kills 6 hidden mutants.
Proposed partial credit
45% repaired tests pass; 20% app bug reported not masked; 20% assertions preserved; 15% mutants killed
R02.07.01.02-T03 Eliminate flakiness from a Cypress suite
Original instruction
Six tests in cypress/e2e/ of the sandbox dashboard fail intermittently (~20% of runs). Find the causes (animation timing, unawaited network calls, shared state, time zone) and fix them in the tests or test setup only.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Cypress · Chrome
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader runs the suite 50 times in CI with randomized order and CPU throttling: 50/50 green. No cy.wait(<number>) added; app source unchanged (diff check). Hidden mutants still detected by the 6 tests.
Proposed partial credit
60% 50/50 stability (linear credit); 20% no fixed waits; 20% tests still kill mutants
R02.07.01.02-T04 Automate an Android onboarding flow with Appium
Original instruction
Write Appium (WebdriverIO) tests for the 'BudgetBee' APK on the API 34 emulator: 4-step onboarding with permission prompts, currency selection, and the settings toggles (dark mode, weekly report). Use accessibility ids where present and UiSelector otherwise.
- Platforms
mac· macOSwin· Windowslin· Linuxand· Android- Applications
- Appium · WebdriverIO · Android Emulator
- Interfaces
mob· Mobile touchterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs the suite on golden APK (must pass twice from clean install) and 5 mutant APKs (skipped step, wrong default currency, toggle not persisted…): kill rate. No driver.pause() calls.
Proposed partial credit
25% passes on golden; 60% mutant kill rate; 15% no fixed pauses
R02.07.01.02-T05 Add XCUITests to a SwiftUI app with accessibility identifiers
Original instruction
fixtures/Recipes (SwiftUI) has no UI tests. Add accessibility identifiers where needed and write XCUITests covering search, favoriting, the recipe detail scaler (servings stepper) and deleting a favorite with swipe.
- Platforms
mac· macOSios· iOS/iPadOS- Applications
- Xcode · XCUITest · iOS Simulator
- Interfaces
mob· Mobile touchdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
xcodebuild test passes on the golden app; hidden mutants (5) each fail ≥1 test. Identifiers must not change visible UI (snapshot SSIM ≥0.99). Swipe-to-delete test must use swipeLeft on the cell.
Proposed partial credit
25% suite passes; 55% mutant kill rate; 20% UI unchanged
R02.07.01.02-T06 Set up visual regression tests with masked dynamic regions
Original instruction
Add Playwright toHaveScreenshot tests for 6 pages of the sandbox analytics app at 1280 and 390 px. Mask or stabilize dynamic regions (clock, live chart, avatars) so the baseline is stable, and commit the baselines.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs the suite 10× against the golden build (all green, so masks work) and against 4 visual-mutant builds (shifted button, color change, font fallback, hidden icon): each must fail. maxDiffPixelRatio must be ≤0.01.
Proposed partial credit
30% stable on golden; 50% visual mutants caught; 20% thresholds strict
R02.07.01.02-T07 Migrate a Selenium Java suite to Playwright TypeScript
Original instruction
Port the 20 Selenium/JUnit tests in legacy-tests/ to Playwright TypeScript with page objects, keeping every scenario and assertion. Run both suites against the sandbox CRM to confirm equal coverage.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Selenium · Playwright · IntelliJ IDEA
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsclick· Point & click- Work product
WP06· Code/engineering- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Grader maps scenarios by test name: 20 ported tests pass on golden; on 10 hidden mutants the Playwright suite kills at least every mutant the Selenium suite kills. Page-object classes exist for the 6 pages.
Proposed partial credit
30% tests ported and passing; 50% mutant parity; 20% page objects
R02.07.01.02-T08 Record a flow with codegen and refactor it into page objects
Original instruction
Use Playwright codegen against the sandbox booking site to record 'search hotel → pick dates → book room as guest', then refactor the recording into a page-object test with assertions on price and confirmation number format.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
click· Point & clickcode· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Test passes on golden and fails on 3 mutants (wrong price, missing confirmation, broken date picker). Page objects for 3 pages exist; no raw coordinate clicks.
Proposed partial credit
30% test passes; 45% mutants caught; 25% page-object structure
R02.07.01.02-T09 Automate a legacy Windows app with no automation ids
Original instruction
The Win32 app 'StockMaster' exposes only partial UIA info (no AutomationIds). Write FlaUI (C#) tests for creating an item, editing its price, and running the 'Low stock' report, locating controls by name, control type and tree position. Tests must survive window moves and DPI 150%.
- Platforms
win· Windows- Applications
- FlaUI · Visual Studio · StockMaster
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
code· Code & commandsclick· Point & clickvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader runs tests at 100% and 150% DPI and with randomized window positions: all pass; 4 mutant builds each fail ≥1 test. No absolute screen coordinates in code (static check).
Proposed partial credit
30% pass at both DPIs; 50% mutant kill rate; 20% no coordinates
R02.07.01.02-T10 Write Maestro flows for three React Native journeys
Original instruction
Write Maestro YAML flows for the 'TripPal' React Native app on the Android emulator: sign in with the test account, create a trip with two stops, and share the itinerary to the clipboard. Use testIDs where present.
- Platforms
mac· macOSwin· Windowslin· Linuxand· Android- Applications
- Maestro · Android Emulator
- Interfaces
mob· Mobile touchterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader runs flows on golden build (pass twice from clean state) and 3 mutants (kill rate). Clipboard content asserted equals itinerary text. No fixed waits (Maestro extendedWaitUntil allowed).
Proposed partial credit
30% passes; 50% mutants killed; 20% clipboard assertion
R02.07.01.02-T11 Test dashboard error states with network mocking
Original instruction
Write Playwright tests for the sandbox metrics dashboard that intercept /api/* and cover: 500 error banner with retry, 10 s timeout spinner then error, empty dataset placeholder, and partial data warning. Inspect the UI to assert visible messages.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Golden build: suite passes. 5 mutants (missing retry, wrong copy, no timeout handling, crash on empty, warning hidden) must each fail. Tests must use page.route, not modify app code.
Proposed partial credit
20% passes; 60% mutants caught; 20% route mocking only
R02.07.01.02-T12 Share page objects across Appium iOS and Android suites
Original instruction
The 'PetVet' app has iOS and Android builds with different element trees. Build one Appium (Python) test suite with platform-specific locator maps and shared page objects covering appointment booking, reschedule and cancel, running on the iOS Simulator and Android emulator.
- Platforms
mac· macOSios· iOS/iPadOSand· Android- Applications
- Appium · iOS Simulator · Android Emulator · XCUITest
- Interfaces
mob· Mobile touchterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader runs on both platforms: golden passes on each; 6 mutants per platform each killed. Shared flow code (page objects) is the same module for both platforms; only locator maps differ (import graph check).
Proposed partial credit
35% iOS suite; 35% Android suite; 20% mutants killed; 10% shared structure
R02.07.01.02-T13 Write component tests for a React design system
Original instruction
Write Playwright component tests (@playwright/experimental-ct-react) for 10 components in packages/ui (Button, Modal, Tabs, Combobox, DatePicker, Toast, Tooltip, Menu, Slider, Table), covering keyboard interaction and visible states.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandskey· Shortcutsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs tests on golden components (all pass) and 20 hidden component mutants (focus trap broken, arrow keys ignored, aria-expanded wrong…): kill rate. Each component needs ≥2 tests.
Proposed partial credit
20% coverage of 10 components; 60% mutant kill rate; 20% keyboard paths
R02.07.01.02-T14 Test a canvas whiteboard with coordinate and pixel assertions
Original instruction
Write Playwright tests for the sandbox canvas whiteboard (no DOM for shapes): draw a rectangle, move it by drag, connect two shapes with an arrow, undo, and export PNG. Assert by exported scene JSON and pixel sampling of the canvas.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright · Excalidraw
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsdrag· Drag & dropvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Golden passes 5× at device scale 1 and 2; 5 mutants (drag offset bug, undo skips, arrow not bound, export missing shape, color wrong) each fail. Tests must compute coordinates from canvas bounding box, not hard-coded page pixels.
Proposed partial credit
25% stable at both scales; 55% mutants caught; 20% coordinate robustness
R02.07.01.02-T15 Test real-time chat with two browser contexts
Original instruction
Write Playwright tests for the sandbox chat app using two users in separate browser contexts: message delivery, typing indicator, read receipts, edit/delete propagation and reconnect after offline. Avoid timing sleeps.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
code· Code & commandsmulti· Cross-app- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Golden passes 20/20 runs; 6 mutants (dropped receipts, stale edits, no reconnect) each fail. Uses two contexts with separate storage states; no waitForTimeout.
Proposed partial credit
25% stable on golden; 55% mutants caught; 20% multi-context hygiene
R02.07.01.02-T16 Shard a Playwright suite in GitHub Actions with traces on failure
Original instruction
Configure .github/workflows/e2e.yml so the 120-test Playwright suite runs sharded across 4 jobs against a service container of the app, uploads traces and HTML reports only on failure, and merges shard reports.
- Platforms
lin· Linux- Applications
- Playwright · GitHub Actions
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader runs the workflow with act: 4 shard jobs, all tests executed exactly once, merged report artifact present. Injected failing test produces a trace.zip artifact; passing run uploads none.
Proposed partial credit
35% sharding correct; 25% service container; 25% failure artifacts; 15% merged report
R02.07.01.02-T17 Write Espresso tests with idling resources for async lists
Original instruction
In fixtures/NewsReader (Android Views), write Espresso tests for the RecyclerView feed: scroll to item 40, open it, bookmark, and verify the bookmarks tab. Register an IdlingResource for the repository's async loads instead of sleeps.
- Platforms
mac· macOSwin· Windowslin· Linuxand· Android- Applications
- Android Studio · Espresso · Android Emulator
- Interfaces
mob· Mobile touchdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandstouch· Touch gestures- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Tests pass 20/20 on emulator with randomized network delay; 4 mutants each killed. Static check: no Thread.sleep / SystemClock.sleep; IdlingResource registered.
Proposed partial credit
30% stable with delays; 50% mutants killed; 20% idling resource used
R02.07.01.02-T18 Separate app regressions from test bugs in a failing nightly run
Original instruction
Nightly run artifacts in fixtures/nightly/ (14 failing Playwright tests, trace.zip files, HTML report) come from the sandbox ERP. Using the trace viewer and the live app, classify each failure as app regression or test defect, fix the test defects, and report regressions in output/regressions.md with evidence.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright · Trace Viewer
- Interfaces
brow· Browserdesk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
vis· Visual judgmentcode· Code & commandsclick· Point & click- Work product
WP06· Code/engineering- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 5 / 5
Proposed verifier
Ground truth: 5 regressions, 9 test defects. Grader scores classification accuracy, then runs fixed tests on golden (pass) and checks the 5 regressions are still asserted (they must fail on the current build). Evidence must name the failing step.
Proposed partial credit
40% classification; 35% fixed tests pass; 15% regressions still detected; 10% evidence
R02.07.01.02-T19 Automate drag-and-drop upload and kanban reordering
Original instruction
Write Playwright tests for the sandbox project app covering drag-and-drop file upload into a card (via DataTransfer), dragging cards between columns and reordering within a column, and keyboard-based reordering (Space, arrows).
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsdrag· Drag & dropkey· Shortcuts- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Golden passes 10/10 runs; 5 mutants (drop index off by one, upload ignores 2nd file, keyboard move broken…) each fail. Assertions read the persisted order via the UI after reload.
Proposed partial credit
25% stable; 55% mutants killed; 20% persistence verified after reload
R02.07.01.02-T20 Build an acceptance suite for 50 user stories
Original instruction
spec/stories.md lists 50 user stories with acceptance criteria for the sandbox HR app 'PeopleDesk'. Write a Playwright suite where each criterion maps to a tagged test (@US-xx), run it against the app, and mark in output/traceability.csv which criteria are covered and whether they pass.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Playwright
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
code· Code & commandsclick· Point & clickvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader checks traceability.csv against tags; suite runs on golden (reported passes must match) and on 25 hidden mutants mapped to stories: kill rate weighted by story. Three criteria are deliberately failing on golden and must be reported as failing.
Proposed partial credit
25% criteria covered; 45% mutant kill rate; 20% failing criteria reported; 10% traceability exact
Mapped research references
No benchmark reference is assigned to this class in the current mapping.