Exploratory playtesting for bugs
Find injected functional bugs by playing.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- test game builds
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R13.03.01.01-T01 Find and report a door that stays locked after picking up its key
Original instruction
A Godot web build of a small dungeon game (build 0.9.3) is open; a bug was injected somewhere in the key-and-door logic. Play the first level, find the defect, and file it in the self-hosted MantisBT project 'Dungeon' with title, area, severity and numbered repro steps from a fresh start.
- Platforms
web· Any OS (web)- Applications
- test game builds · MantisBT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsvis· Visual judgmenttype· Text & forms- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Build telemetry logs when each injected bug's trigger condition fires; a report counts only if its bug fired in the agent's session. Report matched to the bug by area field plus an LLM-assisted match of repro steps to the trigger (rubric, human-audited sample). The game exposes no debug console.
Proposed partial credit
40% bug triggered in session; 30% report matched to the injected bug; 30% repro steps replay the bug (scripted replay)
R13.03.01.01-T02 Find all three injected bugs in a three-room level within 30 minutes
Original instruction
A Unity WebGL build of a 3-room puzzle level is open; 3 functional bugs were injected (collision, state, UI). Explore for at most 30 minutes and file one MantisBT issue per distinct bug with repro steps, expected vs actual behaviour and a screenshot.
- Platforms
web· Any OS (web)- Applications
- test game builds · MantisBT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsvis· Visual judgmenttype· Text & forms- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Recall = injected bugs both triggered (telemetry) and reported (matched); precision penalises reports not matching any injected bug or duplicates. Matching uses area + trigger telemetry + LLM rubric on steps (audited). 30-minute session cap enforced.
Proposed partial credit
60% scaled by recall (a third per bug); 20% precision; 20% screenshots attached to matched reports
R13.03.01.01-T03 Decide whether a quest can be completed and report PASS or FAIL
Original instruction
An RPG test build (Godot, desktop) contains the quest 'Deliver the letter' (pick up letter from the innkeeper, deliver to the blacksmith, return for reward). Play it and record verdict PASS or FAIL in the test-case run in the self-hosted Kiwi TCMS, with the failing step number if FAIL.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- test game builds · Kiwi TCMS
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
key· Shortcutsclick· Point & clickvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Kiwi TCMS execution record: status and failing step compared with the seeded truth (the build has a blocker at step 3: the blacksmith dialogue never offers 'Give letter'). Telemetry must show the agent reached the blacksmith.
Proposed partial credit
30% quest attempted to the blacksmith; 50% correct verdict; 20% correct failing step
R13.03.01.01-T04 Reproduce a reported bug and confirm or reject it
Original instruction
MantisBT issue #12 in project 'Dungeon' reports: 'After dying in the spike room with a potion in inventory, respawn removes the potion.' Using the current web build, follow and vary the steps to confirm or reject the report. Set the issue status to 'confirmed' or 'closed – unable to reproduce' and add a note with what you tried.
- Platforms
web· Any OS (web)- Applications
- test game builds · MantisBT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsvis· Visual judgmenttype· Text & forms- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 2 / 5
Proposed verifier
Mantis DB: issue #12 status equals seeded truth (bug present only when the potion is in slot 1, which the note must mention); telemetry shows the scenario was attempted at least twice with different slots.
Proposed partial credit
40% scenario reproduced in game; 40% correct status; 20% note names the slot condition
R13.03.01.01-T05 Find an item-duplication exploit between bag and chest
Original instruction
An RPG test build (Unity desktop) has an inventory with drag-and-drop between the bag and a storage chest. An injected bug duplicates items under a specific drag sequence. Find it and file a MantisBT report with the exact drag sequence, the item used and how many copies result.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
drag· Drag & dropvis· Visual judgmenttype· Text & forms- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Telemetry: duplication trigger fired (item count increased without pickup). Report matched by area 'Inventory' and replay of the stated drag sequence in an automated input harness reproduces the duplication.
Proposed partial credit
50% duplication triggered; 30% report matched; 20% repro replay succeeds
R13.03.01.01-T06 Find which game state is lost after save and reload
Original instruction
A Godot platformer test build (desktop) supports save/load from the pause menu. An injected bug fails to persist one piece of state. Play, save, quit, reload, and identify what is lost. File a report naming the state, with steps.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsvis· Visual judgmenttype· Text & forms- Work product
WP01· Document- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded truth: collected-gem count resets for gems in the current level only. Telemetry must show a save and a reload; report field 'state lost' must name gems in current level (keyword match plus rubric).
Proposed partial credit
30% save and reload performed; 40% correct lost state identified; 30% report with steps
R13.03.01.01-T07 Locate holes in collision where the player falls through the floor
Original instruction
A Unity 3D test level (desktop build) has 3 injected collision holes in its terrain. Explore and file one report per hole with the position shown on the debug HUD (F3 shows coordinates) within 2 m.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
nav· Pan/zoom/scrollrt· Real-time controltype· Text & forms- Work product
WP10· Annotations- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Each report's coordinates matched to a hole within 2 m; telemetry must show a fall-through event at that hole in the session. Duplicates and false positions reduce precision.
Proposed partial credit
75% scaled by holes found and reported (a third per hole); 25% precision
R13.03.01.01-T08 Find a shop pricing bug that sells items for negative gold
Original instruction
A web RPG test build has a shop with buy/sell and quantity selectors. Under an injected condition, the sell price becomes negative or the buy price underflows. Find the condition and report the exact item, quantity and gold before/after.
- Platforms
web· Any OS (web)- Applications
- test game builds · MantisBT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmenttype· Text & forms- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Telemetry: economy-bug trigger fired (gold delta sign wrong). Report matched by item and quantity range; gold values in report must match the logged transaction.
Proposed partial credit
50% trigger fired; 30% condition reported correctly; 20% gold values exact
R13.03.01.01-T09 Find a softlock caused by pausing during a cutscene
Original instruction
A Godot adventure test build (desktop) plays a cutscene when entering the village. An injected softlock occurs under a specific interaction with the pause menu during the cutscene. Find it and report steps that reliably reproduce the softlock.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsrt· Real-time controlvis· Visual judgment- Work product
WP01· Document- Conditions
interrupt· Interruptions- Proposed difficulty
- 3 / 5
Proposed verifier
Telemetry: softlock state (input ignored > 10 s with game not paused) entered. Scripted replay of reported steps reproduces it at least 2 of 3 times.
Proposed partial credit
50% softlock triggered; 50% replayable steps reported
R13.03.01.01-T10 Find a movement exploit that escapes the level bounds
Original instruction
A 2D platformer test build (web) has an injected bug in jump handling that lets the player leave the level's playable area. Find the input sequence and location, and report them with a short screen recording.
- Platforms
web· Any OS (web)- Applications
- test game builds · MantisBT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsrt· Real-time controlvis· Visual judgment- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Telemetry: player position outside level bounds event. Report includes location and inputs; recording attached (file exists, ≥ 3 s). Replay of inputs reproduces the out-of-bounds event.
Proposed partial credit
50% exploit triggered; 30% replayable report; 20% recording attached
R13.03.01.01-T11 Run a fifteen-item smoke test checklist on a new build
Original instruction
Kiwi TCMS run 'Smoke 1.4.0' lists 15 test cases (launch, new game, save, load, settings apply, audio slider, controller, pause, quit…) for a Unity desktop build. Execute each case and set PASS/FAIL with a comment; 2 cases are seeded to fail.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · Kiwi TCMS
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
click· Point & clickvis· Visual judgmentmulti· Cross-app- Work product
WP02· Records/table- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Kiwi DB: 15 executions with status; accuracy vs seeded truth; telemetry shows each case was actually exercised (e.g. settings change event before marking that case).
Proposed partial credit
70% scaled by correct verdicts; 30% telemetry evidence per case
R13.03.01.01-T12 Verify five claimed fixes in a changelog against the new build
Original instruction
The release notes for build 1.5.1 claim 5 bug fixes (MantisBT #31-#35). Using build 1.5.0 and 1.5.1 side by side, verify each fix. Resolve fixed issues as 'resolved – fixed' and reopen those that are not fixed with a note. One claimed fix is incomplete.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
multi· Cross-appvis· Visual judgmenttype· Text & forms- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Mantis statuses for #31-#35 compared to seeded truth; telemetry from 1.5.1 shows each repro scenario attempted. The incomplete fix reproduces only on the second difficulty, which the reopen note must state.
Proposed partial credit
60% scaled by correct statuses; 20% repro attempted per issue; 20% note identifies the condition
R13.03.01.01-T13 Find a rotation-reset bug in a mobile game build
Original instruction
On the Android emulator, a test build of a mobile puzzle game is installed. An injected bug resets level progress under a device event. Find it and file a report with device model, Android version, the event and steps.
- Platforms
and· Android- Applications
- test game builds · MantisBT
- Interfaces
mob· Mobile touchbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturesvis· Visual judgmenttype· Text & forms- Work product
WP01· Document- Conditions
interrupt· Interruptionsmultiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded truth: rotating the device during the level-complete animation resets progress. Telemetry must show the trigger; report fields (device, OS, event) checked against the emulator profile.
Proposed partial credit
50% trigger fired; 30% event correctly identified; 20% device fields correct
R13.03.01.01-T14 Find untranslated or truncated strings in German menus
Original instruction
A Unity desktop test build is switched to German. Walk through all menus (main, options with 5 tabs, pause, inventory) and report each screen where text is truncated or left in English. There are 4 seeded issues.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
multiapp· Multi-app- Proposed difficulty
- 2 / 5
Proposed verifier
Reports matched by screen + string ID listed in the report or screenshot OCR; recall over 4 seeded issues, precision over all reports. The build's asset bundles are encrypted and the agent has no shell, so string tables cannot be extracted outside the game; build telemetry must show each reported screen was opened in German.
Proposed partial credit
70% scaled by recall; 30% precision
R13.03.01.01-T15 Find a multiplayer desync between two local clients
Original instruction
Two instances of a Godot co-op test build run side by side on one desktop (host and client over localhost). An injected desync makes one action visible on only one client. Control both windows to find it and report which action, on which side, with steps.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appkey· Shortcutsvis· Visual judgment- Work product
WP01· Document- Conditions
multiapp· Multi-apphidden· Hidden/stale state- Proposed difficulty
- 5 / 5
Proposed verifier
Seeded truth: chest opened by the client is not replicated to the host. Telemetry from both instances shows the trigger; report must name the action and the side (rubric plus keywords).
Proposed partial credit
40% trigger exercised on both clients; 40% correct action and side; 20% steps replay
R13.03.01.01-T16 Find why an achievement does not unlock
Original instruction
A desktop test build has 10 achievements visible in its in-game list. 'Pacifist' (finish level 2 without defeating enemies) fails to unlock due to an injected logic bug. Determine the exact condition that blocks it and report it.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
key· Shortcutsvis· Visual judgmenttype· Text & forms- Work product
WP01· Document- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded truth: stunning an enemy counts as a defeat. Telemetry: level 2 finished at least twice with different interactions. Report root-cause field matched by rubric (audited).
Proposed partial credit
30% level 2 completed without kills; 40% blocking condition identified; 30% report quality per rubric
R13.03.01.01-T17 Find a boss-fight position exploit that avoids all damage
Original instruction
A 3D action test build (Unity desktop) has a boss whose attacks cannot reach one spot due to an injected navmesh gap. Find a position from which the boss can be defeated while the player takes no damage. Report the location (HUD coordinates) and approach path.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
nav· Pan/zoom/scrollrt· Real-time controlvis· Visual judgment- Work product
WP10· Annotations- Conditions
multiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Telemetry: boss defeated with 0 damage taken while the player stood within 2 m of the seeded exploit spot for ≥ 80% of the fight; report coordinates within 2 m.
Proposed partial credit
40% exploit spot reached; 40% zero-damage defeat; 20% report
R13.03.01.01-T18 Find a crash and write a minimal reproduction
Original instruction
A Godot desktop test build crashes under an injected input sequence involving the inventory and the map screen. Find a crash, attach the crash log from the user data folder, and reduce the repro to at most 10 steps from the title screen.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
key· Shortcutsvis· Visual judgmentmulti· Cross-app- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Crash signature in the attached log equals the injected crash; scripted replay of the ≤ 10 listed steps crashes the build in 3 of 3 runs. Build telemetry must show the crash fired in the agent's session from GUI input; the Godot build exposes no debug console or remote debugger.
Proposed partial credit
40% crash triggered; 20% log attached; 40% minimal steps reproduce
R13.03.01.01-T19 Find a controller remapping bug that ignores the new binding
Original instruction
A desktop test build supports keyboard rebinding in Options > Controls. An injected bug applies a rebinding in gameplay but not in one other context. Rebind jump to J, find the context where the old key still applies, and report it.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
key· Shortcutsclick· Point & clickvis· Visual judgment- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 2 / 5
Proposed verifier
Seeded truth: the swim state still uses Space. Telemetry shows rebinding done and the swim context entered; report names the context.
Proposed partial credit
30% rebinding applied; 40% swim context found; 30% report
R13.03.01.01-T20 Play a 20-minute vertical slice and report every injected bug
Original instruction
A 20-minute vertical slice of an action-adventure (Unity desktop build) contains 8 injected bugs across combat, UI, saves, audio and collision. Play through it, file one report per distinct bug with steps, expected/actual and evidence, and avoid duplicates.
- Platforms
win· Windowsmac· macOS- Applications
- test game builds · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
key· Shortcutsvis· Visual judgmentav· Audio/video over timetype· Text & forms- Work product
WP01· Document- Conditions
long· Long-horizonmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Recall over 8 bugs (triggered in telemetry and matched report), precision over all reports, duplicate penalty. Matching uses telemetry + area + LLM rubric on steps with human audit of 10% of matches.
Proposed partial credit
60% scaled by recall; 25% precision; 15% evidence attached
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.