Visual glitch & regression detection
Spot clipping, rendering and HUD regressions.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- game captures · engine scenes
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R13.03.01.02-T01 Flag scenes with missing objects between two builds' screenshots
Original instruction
input/captures/ holds 10 pairs of screenshots of the same camera shots from build N−1 and build N of a Unity game. Some build-N shots are missing objects (props, NPCs, effects). In the self-hosted review tool, mark each pair as 'regression' or 'ok' and draw a box around each missing object on the build-N image.
- Platforms
web· Any OS (web)- Applications
- game captures · review tool
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentdrag· Drag & dropclick· Point & click- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Review-tool DB: per-pair verdict F1 vs ground truth; boxes matched to ground-truth missing-object regions at IoU ≥ 0.5. Dynamic elements (clouds, particles) differ in every pair and are not regressions. Terminal and Python are blocked on the VM and the review tool issues no API token, so pairs must be compared by eye in the browser.
Proposed partial credit
50% verdict F1; 50% box recall at IoU ≥ 0.5
R13.03.01.02-T02 Timestamp a wall-clipping glitch in a gameplay video
Original instruction
Open input/run_042.mp4 (60 s gameplay capture, 60 fps) in VLC or the review tool. The character clips through a wall once. Report the start time of the clip-through in answer.json to the nearest 0.5 s and the frame where it is most visible as a PNG.
- Platforms
mac· macOSwin· Windows- Applications
- game captures · VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.json time within ±2 s of ground truth (full credit within ±0.5 s); PNG must perceptually match a frame inside the labelled glitch interval (pHash distance ≤ 6). Terminal, Python and ffmpeg are blocked on the VM, so the glitch must be found and the frame exported by scrubbing in VLC or the review tool.
Proposed partial credit
60% scaled by timestamp error (2 s → 0.5 s); 40% correct frame exported
R13.03.01.02-T03 Check HUD elements against a visual spec
Original instruction
input/hud_spec.md says: health bar top-left, red (#D32F2F ±10 ΔE), ammo counter bottom-right showing '30', minimap top-right 180 px square. Check the 6 screenshots in input/hud/ against the spec and fill hud_check.csv (screenshot, element, pass/fail).
- Platforms
mac· macOSwin· Windows- Applications
- game captures
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmenttype· Text & forms- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
hud_check.csv compared to ground-truth verdicts (18 element checks). Seeded failures: ammo shows '3O', minimap 160 px, bar colour shifted. Terminal and Python are blocked on the VM, so elements must be checked by viewing the images with an image viewer and its colour picker.
Proposed partial credit
50% scaled by correct verdicts on screenshots 1-3 (9 checks); 50% scaled by correct verdicts on screenshots 4-6 (9 checks)
R13.03.01.02-T04 Frame the camera on a reported Z-fighting artifact in Unity
Original instruction
Open the Unity project fixtures/CityBlock in the Unity Editor. Bug report: 'Z-fighting on the north facade sign of building B3 when viewed from the plaza at distance ~25 m.' Move the Scene/Game camera to reproduce it so the flicker is visible, save a screenshot to Captures/zfight.png and save the camera transform in the scene.
- Platforms
win· Windowsmac· macOS- Applications
- Unity
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
geom· Precise geometrynav· Pan/zoom/scrollvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Saved scene: camera position within the plaza region, 20-30 m from the sign, sign inside the view frustum. Screenshot analysed: the sign region shows interleaved stripe pattern (Z-fight detector on two coplanar materials). Fallback VLM check is flagged as such. The agent has no shell, and a scene file written by any process other than the Unity Editor is detected.
Proposed partial credit
40% camera in correct region and distance; 30% sign in view; 30% artifact visible in screenshot
R13.03.01.02-T05 List frames with missing-texture placeholders
Original instruction
input/captures_tex/ holds 30 screenshots from a test pass. Some show the engine's missing-texture placeholder (magenta/black checker) on one or more objects. Write missing_tex.txt listing the filenames that contain it.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- game captures
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Filename set F1 vs ground truth. Seeded hard negatives include a magenta neon sign and a checkered floor. Terminal and Python are blocked on the VM, so screenshots must be inspected in an image viewer rather than colour-matched by script.
Proposed partial credit
50% scaled by F1 from 0 to 0.5; 50% scaled by F1 from 0.5 to 1.0
R13.03.01.02-T06 Timestamp LOD popping events in a fly-through capture
Original instruction
input/flythrough.mp4 (90 s, 60 fps) is a camera fly-through of an open-world scene. Objects visibly 'pop' when switching level of detail. In the review tool, add a marker at each pop (time and a box on the object).
- Platforms
web· Any OS (web)- Applications
- game captures · review tool
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentdrag· Drag & drop- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Markers matched to ground-truth pops within ±0.5 s and box IoU ≥ 0.3: recall and precision. Pops were labelled from engine LOD-switch logs. Terminal, Python and ffmpeg are blocked on the VM and the review tool issues no API token, so pops must be marked while watching in the browser.
Proposed partial credit
60% recall; 25% precision; 15% box quality
R13.03.01.02-T07 Tune shadow bias to remove acne without peter-panning in Godot
Original instruction
Open fixtures/shadow_scene in the Godot editor. The directional light causes shadow acne on the terrain, and raising bias too much detaches shadows from the crates (peter-panning). Adjust the light's shadow bias and normal bias so both artifacts are gone, then save the scene and render Captures/shadow.png from the saved camera.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- Godot
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
geom· Precise geometryvis· Visual judgmentclick· Point & click- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Headless re-render from the saved scene: acne metric (high-frequency shadow noise on lit terrain mask) below threshold and contact-gap metric (unshadowed pixels at crate bases) ≤ 2 px. Edits to .tscn outside the editor detected by editor's scene UID log.
Proposed partial credit
40% acne removed; 40% no peter-panning; 20% capture saved
R13.03.01.02-T08 Report UI elements clipped at two resolutions
Original instruction
Run the Godot desktop build fixtures/menu_build at 1280×720 and 3840×2160 (windowed, set via its Options menu). Visit all 6 menus and report in clipping.csv each UI element that is cut off or overflows the screen at each resolution.
- Platforms
win· Windowsmac· macOS- Applications
- engine scenes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
clipping.csv (menu, element, resolution) F1 vs ground truth (7 seeded issues). Resolution changes must appear in the build's settings log. Terminal and Python are blocked on the VM, so the settings file cannot be edited outside the Options menu and clipping must be judged on screen.
Proposed partial credit
20% both resolutions exercised; 80% scaled by F1
R13.03.01.02-T09 Find T-pose frames in a cutscene capture
Original instruction
input/cutscene.mp4 (2 min, 30 fps) is a cutscene where animation fails at times and characters snap to a T-pose. Report all frame ranges with a T-posed character in tpose.csv (start_frame, end_frame, character).
- Platforms
mac· macOSwin· Windows- Applications
- game captures
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Range overlap with ground truth: temporal IoU per range ≥ 0.5 counts as a match; character name must match. Recall and precision. Terminal, Python and ffmpeg are blocked on the VM, so frames must be inspected in the player rather than analysed by script.
Proposed partial credit
60% recall; 25% precision; 15% character names
R13.03.01.02-T10 Detect missing particle effects between two builds
Original instruction
input/fx_pairs/ holds 12 short clips (3 s) of the same scripted moments in build N−1 and N (explosions, spells, footsteps dust). Mark in fx_check.csv which build-N clips are missing an effect present in N−1, naming the effect.
- Platforms
mac· macOSwin· Windows- Applications
- game captures
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgment- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
fx_check.csv verdict F1 vs ground truth; effect name matched against a synonym list. Particle randomness differs between builds even when correct. Terminal, Python and ffmpeg are blocked on the VM, so clips must be compared in the player rather than analysed by script.
Proposed partial credit
70% verdict F1; 30% effect names
R13.03.01.02-T11 Find a material with an inverted normal map in Unreal
Original instruction
Open the Unreal Editor project fixtures/Courtyard. Lighting on one wall material looks wrong (bumps appear as dents). Use buffer visualisation (World Normal) and the material editor to find the material whose normal map has an inverted green channel, fix it by setting 'Flip Green Channel' on the texture, and save.
- Platforms
win· Windowsmac· macOS- Applications
- Unreal Engine
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentclick· Point & clicknav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Saved .uasset metadata parsed (via a commandlet in the harness): only the seeded texture has bFlipGreenChannel = true; no other assets modified (package hash diff). Python editor scripting plugin is disabled.
Proposed partial credit
40% correct material identified (texture changed); 40% fix applied; 20% no other assets changed
R13.03.01.02-T12 Measure a tonemapping colour regression on reference patches
Original instruction
input/tonemap/ has captures of an in-game colour chart from build N−1 and N (SDR). For each of the 24 patches, measure mean colour in a patch-centre region using an image editor's colour picker or info panel and list patches with ΔE2000 > 3 between builds in tonemap.csv.
- Platforms
mac· macOSwin· Windows- Applications
- game captures · Photoshop · GIMP
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
vis· Visual judgmentgeom· Precise geometrytype· Text & forms- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
tonemap.csv patch set vs ground truth computed from the images (patches with ΔE2000 > 3); reported ΔE values within ±0.5. Terminal and Python are blocked on the VM, so sampling must be done in the editor; accuracy of manual sampling is what is measured.
Proposed partial credit
60% patch set F1; 40% ΔE values within tolerance
R13.03.01.02-T13 Locate flickering geometry in a long capture
Original instruction
input/walk_2min.mp4 (2 min, 60 fps) contains intermittent single-frame flicker of a distant building (visibility toggling). Report every flicker event's frame number in flicker.csv.
- Platforms
mac· macOSwin· Windows- Applications
- game captures · VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Frame numbers matched to ground truth within ±2 frames; recall and precision. Events last 1-2 frames, so frame stepping is required. Terminal, Python and ffmpeg are blocked on the VM, so frames must be stepped through in VLC rather than analysed by script.
Proposed partial credit
70% recall; 30% precision
R13.03.01.02-T14 Flag overlapping text in localized menu screenshots
Original instruction
input/loc_shots/ has 40 screenshots of the same 10 menus in 4 languages (EN, DE, JA, AR). Flag every screenshot where text overlaps other text or UI elements, or runs outside its button, in loc_flags.csv (file, element).
- Platforms
mac· macOSwin· Windows- Applications
- game captures
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
loc_flags.csv file+element matched to ground truth (9 seeded issues); F1. Right-to-left layout in AR is correct by design and is a hard negative. Terminal and Python are blocked on the VM, so screenshots must be inspected in an image viewer rather than analysed by OCR scripts.
Proposed partial credit
50% scaled by F1 from 0 to 0.5; 50% scaled by F1 from 0.5 to 1.0
R13.03.01.02-T15 Reproduce a terrain-clipping report in Godot and file it with the camera transform
Original instruction
Bug report: 'Camera clips into the hill near the windmill when orbiting at max zoom.' Open fixtures/valley in the Godot editor, run the scene, reproduce the clipping with the in-game orbit camera, capture Captures/clip.png and file a MantisBT note with the camera position and rotation shown in the debug overlay.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- Godot · MantisBT
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
geom· Precise geometrynav· Pan/zoom/scrollvis· Visual judgment- Work product
WP10· Annotations- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Reported camera position inside the terrain mesh (ray test in harness) near the windmill (≤ 30 m); screenshot shows terrain interior (backface ratio metric). Note added to the issue. Terminal and Python are blocked on the VM, and the harness logs debug-overlay transforms per frame, so the reported transform must be one the run actually reached.
Proposed partial credit
40% clipping reproduced; 30% transform reported correctly; 30% screenshot and note filed
R13.03.01.02-T16 Identify stutter from duplicated frames in a capture
Original instruction
input/pan_60fps.mp4 is a 30 s capture of a smooth camera pan recorded at 60 fps. Frame pacing problems show up as duplicated or skipped frames. Report the time ranges with stutter in stutter.csv (start_s, end_s).
- Platforms
mac· macOSwin· Windows- Applications
- game captures · VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
av· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Ranges matched to ground truth (derived from engine frame-time log) with temporal IoU ≥ 0.5; recall and precision. Terminal, Python and ffmpeg are blocked on the VM, so duplicated frames must be found by stepping in the player rather than by frame hashing.
Proposed partial credit
70% recall; 30% precision
R13.03.01.02-T17 Approve or reject 25 visual-regression diffs
Original instruction
A self-hosted visual-regression dashboard (reg-suit style) shows 25 diffs between golden images and the new build, each with a changelog note. Approve diffs that match an intended change in the changelog and reject the rest as regressions with a one-line reason.
- Platforms
web· Any OS (web)- Applications
- visual regression dashboard
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentclick· Point & clicktype· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Dashboard DB: approve/reject per diff vs ground truth; accuracy. Reasons are recorded; rejection reasons spot-checked by rubric. The dashboard issues no API token and accepts verdicts only from the CSRF-bound browser session; Terminal and Python are blocked, so diffs cannot be scored by script.
Proposed partial credit
85% scaled by verdict accuracy; 15% reasons present
R13.03.01.02-T18 Check HUD safe-area violations on notched device profiles
Original instruction
On the Android emulator, install fixtures/hud_test.apk and run it on 3 device profiles (Pixel 7, a 20:9 notch device, a tablet). Screenshot the HUD in each and report in safe_area.csv any element overlapping the cutout or rounded corners.
- Platforms
and· Android- Applications
- engine scenes
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
touch· Touch gesturesvis· Visual judgment- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
safe_area.csv (device, element) F1 vs ground truth; screenshots for each profile must exist with the right resolution. Python and image-analysis tools are not installed on the agent VM, so overlaps with the cutout and corners must be judged by eye.
Proposed partial credit
30% three profiles run with screenshots; 70% F1
R13.03.01.02-T19 Use the Unity Frame Debugger to fix an object that disappears
Original instruction
In Unity project fixtures/Shop, the glass display case vanishes when viewed in front of the skybox. Use the Frame Debugger to find the draw call ordering problem, fix the material's render queue or transparency settings in the Inspector so it renders correctly, and save.
- Platforms
win· Windowsmac· macOS- Applications
- Unity
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentclick· Point & clicknav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Headless player render from the saved project: case visible (pixel mask overlap ≥ 0.9 vs reference). Material YAML diff limited to the glass material's queue/blend fields; other assets unchanged. The agent has no shell, and a material file written by any process other than the Unity Editor is detected.
Proposed partial credit
30% correct material identified; 50% case renders correctly; 20% no unrelated asset changes
R13.03.01.02-T20 Triage 120 screenshot pairs for regressions with noisy dynamic content
Original instruction
A review tool lists 120 screenshot pairs (golden vs nightly) across 12 levels; weather, crowds and time of day vary between captures. Mark each pair 'regression' or 'ok' and, for regressions, box the region and pick a category (missing, wrong material, lighting, UI, geometry).
- Platforms
web· Any OS (web)- Applications
- game captures · review tool
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentdrag· Drag & dropclick· Point & click- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Verdict precision/recall (target ≥ 0.9 each); boxes IoU ≥ 0.5; category accuracy. 21 seeded regressions; dynamic noise is the hard negative. Terminal and Python are blocked on the VM and the review tool issues no API token, so pairs must be judged by eye in the browser.
Proposed partial credit
50% verdict F1; 30% box recall; 20% category accuracy
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.