Exploratory UI testing & bug reports
Find, reproduce and write up defects in an app by using it.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- any app · bug tracker
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R02.07.01.01-T01 Explore a sandbox web shop checkout and file defects in Jira
Original instruction
The staging shop 'Shoply' (test accounts in fixtures/accounts.md) has a new cart and checkout. Explore it for defects in cart, promo codes, shipping and payment (test card only). File each defect in the Jira sandbox project SHOP with title, steps, expected, actual, severity and a screenshot.
- Platforms
web· Any OS (web)- Applications
- Jira · Shoply staging
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsvis· Visual judgment- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Grader matches Jira issues to 6 seeded bugs by URL, element accessible name and expected/actual values; per-bug credit. Unmatched issues cost 5% each. Screenshots must show the defect (VLM check). Seeded bugs are client-side only, so no API reveals them; agent has no Jira API token.
Proposed partial credit
70% seeded bugs found (6 equal shares); 15% fields complete; 15% screenshot evidence; false reports deduct 5 pts each
R02.07.01.01-T02 Hunt for crashes in an Android notes app and attach logcat traces
Original instruction
On the Pixel 7 API 34 emulator, the test build of 'QuickNotes' has crash bugs. Use the app (long text, emoji, attachments, rotation, sharing into it) to find crashes, and open a GitHub issue in the sandbox repo for each with minimal steps and the logcat stack trace.
- Platforms
and· Android- Applications
- QuickNotes · GitHub Issues · Android Emulator
- Interfaces
mob· Mobile touch- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturestype· Text & formsvis· Visual judgment- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Each issue's stack trace is matched to 4 seeded exceptions by top-frame signature; steps are replayed by a grader script mapped from the reported action list to UiAutomator calls (must crash). Duplicate issues earn nothing. APK is R8-obfuscated with no source or mapping on the VM, and adb is wrapped to allow logcat only (am, monkey, input blocked), so crashes must come from UI use.
Proposed partial credit
60% seeded crashes found (15 pts each); 25% steps replay to the crash; 15% traces attached
R02.07.01.01-T03 Turn a vague 'total is sometimes wrong' report into an exact repro
Original instruction
Customer ticket CS-881 says the order total in the sandbox billing portal is 'sometimes wrong'. Find the exact conditions that trigger it, reduce to minimal steps, and update the ticket in Linear with the repro, expected vs actual amounts, and the affected scope.
- Platforms
web· Any OS (web)- Applications
- Linear · Billing portal sandbox
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Seeded condition: EUR currency + percentage coupon + quantity edit after coupon. Grader parses the Linear issue: the three conditions must be named; steps ≤8 and replay to the wrong total; expected/actual amounts exact. LLM rubric (10%) only for clarity. Portal JS ships minified with source maps withheld and no backend/DB access; no Linear API key is issued, so the issue is edited in the browser.
Proposed partial credit
45% trigger conditions; 25% minimal steps replay; 20% amounts exact; 10% clarity rubric
R02.07.01.01-T04 Find layout regressions between two builds at three viewports
Original instruction
Builds A (production) and B (release candidate) of the marketing site run at two staging URLs. Compare all 8 pages at 1440, 768 and 375 px and report each layout regression in B as a Linear issue with page, viewport, a screenshot and the element affected.
- Platforms
web· Any OS (web)- Applications
- Linear · Chrome
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentnav· Pan/zoom/scrollclick· Point & click- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded: 5 regressions (overlap, clipped text, wrong breakpoint, missing image, misaligned footer), matched by page, viewport and element selector or box IoU ≥0.5 with the seeded region. Changes listed in release notes are not bugs; reporting them counts as false positives. Agent VM has no headless browser or Playwright and no Linear API key, so issues are filed from the browser session.
Proposed partial credit
75% seeded regressions (15 pts each); 15% evidence screenshots; 10% no intentional changes reported
R02.07.01.01-T05 Test a macOS app in German and Japanese for localization defects
Original instruction
Run the test build of 'Tally' (macOS) with -AppleLanguages de and ja. Inspect every screen and menu for untranslated strings, truncation, wrong date/number formats and layout breaks. Record findings in output/l10n_findings.csv (screen, locale, element, issue_type, observed).
- Platforms
mac· macOS- Applications
- Tally (test build)
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
vis· Visual judgmentclick· Point & clicknav· Pan/zoom/scroll- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
CSV matched to 9 seeded defects by screen, locale, element and issue_type; precision and recall computed; pass needs recall ≥0.8 and precision ≥0.7. The strings catalog is stripped from the bundle and System Events/JXA UI scripting is disabled for the agent, so defects must be found by viewing the UI.
Proposed partial credit
60% recall; 25% precision; 15% schema-valid CSV
R02.07.01.01-T06 Find validation bugs in a legacy Windows invoice entry form
Original instruction
On the Windows VM, the WinForms app 'InvoicePro' has an invoice entry screen. Explore field validation (dates, VAT rates, negative quantities, rounding, 255+ char text, paste) and report defects in the Azure DevOps sandbox board as Bugs with steps and expected/actual.
- Platforms
win· Windows- Applications
- InvoicePro · Azure DevOps
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
type· Text & formsclick· Point & clickvis· Visual judgment- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Azure DevOps work items are matched to 7 seeded validation bugs by field name and triggering input class; repro steps replayed via FlaUI from the stated inputs. False positives −4% each. No Azure DevOps PAT is issued and the app has no API; UI Automation tools (FlaUI, PowerShell UIA) are blocked for the agent.
Proposed partial credit
70% seeded bugs (10 pts each); 20% replayable steps; 10% correct severity per rubric table
R02.07.01.01-T07 Triage 25 incoming GitHub issues by reproducing each
Original instruction
The sandbox repo 'taskboard' has 25 new issues. Reproduce each against the running app, then label confirmed, cannot-reproduce or needs-info, add a 'duplicate of #N' comment and close for duplicates, and add the component label (ui, sync, auth, export).
- Platforms
web· Any OS (web)- Applications
- GitHub Issues · Taskboard app
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsmulti· Cross-app- Work product
WP08· App/system state- Conditions
multiapp· Multi-applong· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Ground-truth table: per issue, expected status label, duplicate target and component. Grader reads labels/comments via read-only API: accuracy on status, duplicates and component. Closing non-duplicates costs 5% each.
Proposed partial credit
45% status labels; 25% duplicates linked and closed; 20% component labels; 10% no wrong closures
R02.07.01.01-T08 Verify ten 'fixed' Jira tickets on a new build
Original instruction
Jira sandbox filter 'Ready for QA' lists 10 tickets fixed in build 2.4.1 of the web app. Verify each fix on the 2.4.1 staging URL; move verified ones to Done, and reopen the others with a comment stating what still fails.
- Platforms
web· Any OS (web)- Applications
- Jira · Web app staging
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsvis· Visual judgment- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 2 / 5
Proposed verifier
Seeded: 7 truly fixed, 3 still broken. Grader reads transitions and comments: correct final status per ticket; reopened tickets' comments must mention the failing step keyword from the ticket's repro. No Jira API token is issued and the app's DB/logs are not reachable, so each fix must be checked and transitioned in the browser.
Proposed partial credit
70% correct status per ticket (7 pts each); 30% reopen comments specific
R02.07.01.01-T09 Find collision bugs in a 2D platformer test build
Original instruction
Play the test build of 'Lumen' (level 1-1 to 1-3) on the desktop with keyboard controls. Find places where the player falls through floors, clips into walls or gets stuck, and log each in output/collision_bugs.json with level, x/y from the debug overlay and the action that triggers it.
- Platforms
mac· macOSwin· Windows- Applications
- Lumen (test build)
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
rt· Real-time controlkey· Shortcutsvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Grader matches entries to 6 seeded collision faults by level and position (within 32 px) and trigger action; each reported entry is replayed by a scripted input bot from the stated position: must reproduce. Level data ships encrypted, the debug console/API is disabled and memory readers are blocked on the VM, so faults must be found by playing.
Proposed partial credit
60% seeded faults (10 pts each); 30% entries reproduce; 10% schema
R02.07.01.01-T10 Test an iOS app's behavior under interruptions
Original instruction
In the iOS Simulator, test 'FitTrack' workout recording against interruptions: backgrounding, lock screen, low-memory warning, rotation, a simulated incoming call and changed location permission. Report defects in output/interruption_bugs.md with steps, expected and actual.
- Platforms
ios· iOS/iPadOS- Applications
- FitTrack · iOS Simulator
- Interfaces
mob· Mobile touch- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturesvis· Visual judgment- Work product
WP01· Document- Conditions
interrupt· Interruptions- Proposed difficulty
- 4 / 5
Proposed verifier
Seeded: 5 state-loss defects. Grader parses the markdown into entries (fixed headings) and matches by interruption type and screen; each matched entry's steps are replayed via XCUITest + simctl. LLM rubric only for clarity (10%). Release build has no source or dSYM; simctl and XCUITest are blocked for the agent, so interruptions come via Simulator menus and gestures.
Proposed partial credit
65% seeded defects (13 pts each); 25% replayable; 10% clarity rubric
R02.07.01.01-T11 Boundary-test a signup form for validation defects
Original instruction
Test the signup form of the sandbox web app: email, password rules, date of birth, phone, country/postcode and terms checkbox. List every defect in output/signup_bugs.csv (field, input, expected, actual).
- Platforms
web· Any OS (web)- Applications
- Signup sandbox app
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & click- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
6 seeded defects (e.g., accepts 31 Feb, password rule off-by-one). CSV rows matched by field and an input that triggers the same equivalence class; grader re-enters the input via Playwright to confirm. Precision penalty for unmatched rows. The signup endpoint accepts only requests carrying the page's CSRF token and browser session cookie, and the bundle has no source maps, so inputs must be entered in the form.
Proposed partial credit
75% seeded defects (12.5 pts each); 15% inputs reproduce; 10% precision
R02.07.01.01-T12 Find browser-specific bugs across Chrome, Firefox and Safari
Original instruction
Test the sandbox scheduling app in Chrome, Firefox and Safari (WebKit on macOS). Find defects that appear in only one or two browsers (date pickers, drag-to-reschedule, clipboard, layout) and file them in Linear tagged with the affected browsers.
- Platforms
mac· macOSweb· Any OS (web)- Applications
- Chrome · Firefox · Safari · Linear
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clickvis· Visual judgment- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
3 seeded browser-specific bugs; matched by feature and exact affected-browser set. Reports tagging the wrong browser set get half credit. Grader replays steps with Playwright in each engine. App bundles are minified without source maps and no Linear API key is issued, so issues are filed from a browser session.
Proposed partial credit
60% seeded bugs; 25% correct browser sets; 15% replayable steps
R02.07.01.01-T13 Run a timed exploratory session with a charter and report
Original instruction
Run a 45-minute session-based exploratory test of the 'Export' feature in the desktop app 'ChartLab' (CSV, PNG, PDF exports). Write output/session.md with charter, areas covered, test notes, bugs found (steps/expected/actual) and issues/questions, using the provided SBTM template.
- Platforms
mac· macOSwin· Windows- Applications
- ChartLab
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Bugs section matched to 4 seeded export defects (wrong DPI, truncated CSV, PDF fonts, locale decimals). Template sections must be present (parser). Coverage claims checked against the app's telemetry log of visited dialogs. LLM rubric for notes quality (15%).
Proposed partial credit
50% seeded bugs; 20% coverage claims match telemetry; 15% template complete; 15% rubric
R02.07.01.01-T14 Rewrite eight vague bug reports into reproducible ones
Original instruction
Eight bug reports in the Bugzilla sandbox (product 'PhotoShelf' web app) are vague ('upload broken', 'weird colors'). Reproduce each in the app and edit it with precise steps, environment, expected and actual; resolve as WORKSFORME the ones you cannot reproduce after trying.
- Platforms
web· Any OS (web)- Applications
- Bugzilla · PhotoShelf
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 2 / 5
Proposed verifier
6 of 8 are reproducible by seeded triggers. Grader reads each bug: steps mention the trigger condition (keyword table) and replay via Playwright; the 2 non-reproducible ones must be WORKSFORME; no reproducible bug closed. No Bugzilla API key is issued and PhotoShelf source/DB are not reachable, so bugs are reproduced and edited in the browser.
Proposed partial credit
60% reproducible reports rewritten (10 pts each); 25% correct WORKSFORME; 15% fields complete
R02.07.01.01-T15 Locate UI freezes in an Electron app and report with a profile
Original instruction
Users say 'Kanbanize Desktop' (test build) freezes randomly. Find which user actions block the UI for over 2 s, capture a DevTools performance profile for each, and file reports in output/freezes/ (one folder per freeze with steps.md and profile.json).
- Platforms
mac· macOSwin· Windows- Applications
- Kanbanize Desktop · Chrome DevTools
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
3 seeded freezes. Grader replays steps with Playwright _electron and measures long tasks: must exceed 2 s; profile.json must contain the seeded function in the top self-time frames. Unrelated folders cost 5% each.
Proposed partial credit
45% freezes found; 30% steps reproduce freeze; 25% profiles identify hot function
R02.07.01.01-T16 Find stale-state bugs between views in a CRM sandbox
Original instruction
In the sandbox CRM, edits made in one place sometimes do not appear elsewhere (list vs detail vs kanban vs report). Explore contacts, deals and tasks, and report each propagation bug in output/stale_bugs.md with the edit made, where it was missing, and how to refresh to see it.
- Platforms
web· Any OS (web)- Applications
- CRM sandbox
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsvis· Visual judgment- Work product
WP01· Document- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 3 / 5
Proposed verifier
4 seeded caching bugs; matched by entity type, source view and stale view. Grader replays the stated edit via Playwright and checks the stale view shows old data. Server data is correct, so only the UI reveals the bugs.
Proposed partial credit
70% seeded bugs; 20% replay confirms; 10% refresh workaround correct
R02.07.01.01-T17 Test offline sync in an Android app on a throttled network
Original instruction
On the API 34 emulator, test 'FieldLog' (offline-first form app) while toggling airplane mode and throttling the network via the emulator controls. Create, edit and delete records offline and online; report sync defects with exact sequences in output/sync_bugs.json.
- Platforms
and· Android- Applications
- FieldLog · Android Emulator
- Interfaces
mob· Mobile touch- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturestype· Text & forms- Work product
WP01· Document- Conditions
interrupt· Interruptionshidden· Hidden/stale state- Proposed difficulty
- 3 / 5
Proposed verifier
3 seeded sync bugs (lost edit, duplicate on retry, delete resurrected). Grader replays each reported sequence with UiAutomator + emulator console network commands and checks the backend record state. Precision penalty for non-reproducing entries. APK is R8-obfuscated with no source, and the backend DB and API docs are not reachable, so bugs are found by using the app with the emulator controls.
Proposed partial credit
60% seeded bugs; 30% sequences reproduce; 10% schema-valid JSON
R02.07.01.01-T18 Probe role permissions in an admin app for UI authorization bugs
Original instruction
The sandbox admin portal has roles Viewer, Editor and Admin (test accounts in fixtures/roles.md). Using each account, check what each role can see and do in the UI against spec/permissions.md, and report every mismatch in output/rbac_findings.csv. Do not modify data belonging to the 'prod-mirror' tenant.
- Platforms
web· Any OS (web)- Applications
- Admin portal sandbox
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP02· Records/table- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
5 seeded mismatches (e.g., Viewer sees Delete and it works). Rows matched by role, action and screen. Grader checks the audit log: any write to tenant prod-mirror fails the task. Precision penalty for spec-consistent rows. No API token is issued, so role checks must be done by signing in as each role in the browser.
Proposed partial credit
70% mismatches found; 15% precision; 15% no forbidden writes (gate)
R02.07.01.01-T19 Run a full release-candidate exploratory pass on a PM app
Original instruction
Release candidate 3.0 of the sandbox project-management app 'Planr' (boards, Gantt drag, comments, notifications, CSV import, settings) must be signed off. Explore it end to end and file all defects in the Jira sandbox project PLANR with severity, then add a sign-off comment on PLANR-1 recommending go or no-go.
- Platforms
web· Any OS (web)- Applications
- Planr · Jira
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & formsvis· Visual judgment- Work product
WP01· Document- Conditions
long· Long-horizonmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
15 seeded bugs (3 blocker, 5 major, 7 minor). Grader matches issues by feature area, element and trigger; severity scored against seeded levels. The recommendation must be 'no-go' (blockers exist) citing ≥1 blocker key. Planr ships minified without source maps and no Jira API token is issued, so bugs must be found and filed through the browser.
Proposed partial credit
50% bugs found (weighted by severity); 20% severity correct; 15% replayable steps; 15% go/no-go with blocker cited
R02.07.01.01-T20 Find a reliable repro for duplicate orders from double submit
Original instruction
Support reports occasional duplicate orders in the sandbox ordering app. Find the exact user interaction and timing that creates duplicates (it involves the Submit button and slow network), reduce it to a reliable repro, and document it in output/dup_order.md with the success rate over 10 tries.
- Platforms
web· Any OS (web)- Applications
- Ordering app sandbox · Chrome DevTools
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickrt· Real-time controlvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Seeded race: second click within the 300–900 ms window after the first under 'Slow 3G' throttling. Grader parses the doc for interaction, timing window and throttle setting, then replays 10× via Playwright: ≥8/10 duplicates. Reported rate within ±20% of measured. The order API accepts only requests carrying the page's CSRF token and browser session cookie, and the bundle has no source maps, so the race must be found via the UI.
Proposed partial credit
35% correct trigger and timing window; 35% replay reliability; 20% throttle condition; 10% rate accuracy
Mapped research references
No benchmark reference is assigned to this class in the current mapping.