Policy rollout review & failure labeling
Review robot policy rollouts and classify failures with timestamps.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- LeRobot · Isaac Lab · MuJoCo viewer
- Class GUI-need label
HY· Hybrid
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R07.09.02.02-T01 Label success or failure for ten pick-and-place rollout videos
Original instruction
Watch the 10 rollout videos in fixtures/rollouts_pick/ (Franka, 'put the mug on the coaster') and write output/labels.csv with episode_id and outcome (success/failure). Success means the mug rests upright on the coaster at the final frame.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo viewer · VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader compares to ground truth from privileged sim logs; accuracy over 10 episodes. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
Proposed partial credit
50% episodes 1-5 correctly labeled; 50% episodes 6-10 correctly labeled (10 pts each).
R07.09.02.02-T02 Classify failure modes and onset times for 30 failed episodes
Original instruction
For the 30 failed episodes in fixtures/fail_set_A/ (videos + joint logs), label each with one mode from input/taxonomy.md (missed grasp, slip in transport, collision, wrong object, timeout) and the onset time (s) when the failure became inevitable. Write output/failures.csv.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- LeRobot · VLC · Python
- Interfaces
desk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentcode· Code & commands- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 3 / 5
Proposed verifier
Grader compares to labels derived from privileged contact/grasp logs: mode accuracy and onset within +/-0.5 s. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
Proposed partial credit
60% mode accuracy (pro-rata); 40% onset within tolerance (pro-rata).
R07.09.02.02-T03 Find low-quality teleop episodes with the LeRobot dataset visualizer
Original instruction
Open the 50-episode dataset fixtures/aloha_insert_v2 in the LeRobot dataset visualizer. Find episodes with frozen camera streams (>=1 s identical frames), action spikes (jumps >0.3 rad in one step) or missing gripper motion, and list them with the issue type in output/bad_episodes.csv.
- Platforms
lin· Linuxmac· macOS- Applications
- LeRobot
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
av· Audio/video over timevis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader has planted issues in 7 episodes; F1 over (episode, issue_type) pairs.
Proposed partial credit
50% recall of planted (episode, issue) pairs; 50% precision of reported pairs.
R07.09.02.02-T04 Evaluate an Isaac Lab checkpoint over 100 seeds and label failures
Original instruction
Run fixtures/checkpoints/lift_cube_v3.pt in Isaac Lab's Franka lift task for seeds 0-99 with video recording, report the success rate, and label each failed seed with a mode (no grasp, drop, overshoot, timeout) in output/eval.csv. Use the provided play script; do not retrain.
- Platforms
lin· Linux- Applications
- Isaac Lab
- Interfaces
term· Terminaldesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsav· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader reruns the deterministic evaluation: success rate exact; failure-mode labels compared to privileged per-seed event logs (accuracy).
Proposed partial credit
30% evaluation runs with correct seeds; 20% success rate exact; 50% failure mode accuracy.
R07.09.02.02-T05 Locate the fall and first contact loss in a humanoid recording
Original instruction
Open fixtures/humanoid_walk.rrd in the Rerun viewer. Find the time the humanoid falls (pelvis height <0.4 m) and which foot first lost stable contact before the fall, and record both in output/answer.json (time in s, foot = left/right).
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Rerun
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentav· Audio/video over time- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader computes from the recording's hidden ground-truth channels: fall time within 0.1 s, foot exact.
Proposed partial credit
50% fall time; 50% foot.
R07.09.02.02-T06 Compare two policies seed by seed and explain B's regressions
Original instruction
Policies A and B were run on the same 40 seeds (fixtures/ab_eval/{A,B}/ videos and logs). List seeds where A succeeds and B fails, and label B's failure mode per input/taxonomy.md with onset time. Write output/regressions.csv.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- LeRobot · Python · VLC
- Interfaces
desk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentcode· Code & commands- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader compares the regression seed set (F1) and modes/onsets against privileged logs (onset +/-0.5 s). Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
Proposed partial credit
40% regression set F1; 40% mode accuracy; 20% onsets.
R07.09.02.02-T07 Report the step where an object first hits the floor
Original instruction
In MuJoCo simulate, load fixtures/shelf_drop.xml and play the recorded trajectory fixtures/traj_17.mjtraj; report the simulation step at which the box first touches the floor in output/answer.txt.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo viewer
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader computes first floor contact from the trajectory; step within +/-2. The agent VM has no terminal or MuJoCo Python bindings and contact data is withheld, so the step must be found by scrubbing playback in simulate.
Proposed partial credit
50% step within 10 of truth; 50% step within 2 of truth.
R07.09.02.02-T08 Annotate failure spans across 200 rollouts and build a mode histogram
Original instruction
Import the 200 rollout videos in fixtures/bin_pick_200/ into the provided CVAT project, annotate a temporal span for every failure event with its mode label, export the annotations, and write output/histogram.csv plus one representative clip per mode (output/clips/<mode>.mp4).
- Platforms
web· Any OS (web)- Applications
- CVAT · ffmpeg
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
av· Audio/video over timevis· Visual judgmentdrag· Drag & drop- Work product
WP10· Annotations- Conditions
long· Long-horizonmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Grader matches exported spans to privileged-event spans: temporal IoU >=0.5 counts as match; label accuracy on matches; histogram from labels must equal counts; each clip must contain a true event of its mode.
Proposed partial credit
40% span F1; 30% label accuracy; 15% histogram; 15% clips valid.
R07.09.02.02-T09 Segment subtask phases in fifteen demonstration videos
Original instruction
In Label Studio (video timeline), segment each of the 15 videos in fixtures/demos/ into phases reach, grasp, lift, transport and place, following input/phase_rules.md, and export JSON to output/phases.json.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timedrag· Drag & dropvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader compares to reference segmentation from gripper/object state logs: mean per-phase temporal IoU; full credit at >=0.7. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
Proposed partial credit
50% for mean temporal IoU up to 0.35 (pro-rata); 50% for mean temporal IoU from 0.35 to 0.7 (pro-rata).
R07.09.02.02-T10 Flag rollouts with self-collisions or joint-limit violations
Original instruction
Review the 60 rollouts in fixtures/bimanual_eval/ (videos + joint_state CSVs + URDF limits) and list every episode with a self-collision or joint-limit violation, its time, the joints/links involved and severity per input/severity.md in output/safety.csv.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Isaac Lab · Python · VLC
- Interfaces
desk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsav· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader: joint-limit events derivable from CSV (exact match required); self-collisions from hidden contact logs (visual-only for agent): F1 over episodes, time +/-0.3 s, severity accuracy.
Proposed partial credit
35% joint-limit events; 35% self-collision events; 15% times; 15% severity.
R07.09.02.02-T11 Find lucky successes where the object reached the goal by accident
Original instruction
All 25 episodes in fixtures/push_success/ are flagged success by the env. Watch them and mark episodes where the goal was reached by bounce, knock-over or after leaving the table, rather than a controlled push, in output/lucky.csv with the time of the uncontrolled event.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo viewer · VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader has 6 planted lucky episodes (from contact logs); F1 over episodes and event time within 0.5 s. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
Proposed partial credit
70% episode F1; 30% event times.
R07.09.02.02-T12 Mark where real-robot runs diverge from their sim twins
Original instruction
fixtures/sim2real/ holds 12 pairs of sim and real videos of the same policy and initial state. For each real episode, label whether it diverges from sim and, if so, the phase (approach, grasp, lift, place) and time where divergence starts, in output/divergence.csv.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- VLC · LeRobot
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader compares to expert reference labels (two-annotator consensus): divergence flag accuracy, phase accuracy, time within 1 s. Sim state and robot logs are withheld from the agent; only the video pairs are provided, so divergence must be judged by viewing them.
Proposed partial credit
40% divergence flags; 35% phase; 25% time.
R07.09.02.02-T13 Spot reward-hacking behavior in training-time rollouts
Original instruction
Rollouts from an Isaac Lab peg-insert run (fixtures/peg_hack/, 40 videos + reward logs) show rising reward. Identify the episodes where reward is earned by an exploit (e.g., peg clipping through the board or tilting the socket) and describe the exploit in output/hack_report.md with episode ids and times.
- Platforms
lin· Linux- Applications
- Isaac Lab · VLC · Python
- Interfaces
desk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentcode· Code & commands- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Grader: planted exploit episodes (from physics penetration logs) F1 >=0.8 for full credit; exploit type must match one of 2 planted mechanisms (keyword rubric); times within 0.5 s. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
Proposed partial credit
50% episode F1; 30% exploit mechanisms named; 20% times.
R07.09.02.02-T14 Compute rollout metrics from labeled logs (control pair of T02)
Original instruction
Control pair of T02: given fixtures/fail_set_A/labels_gt.parquet with per-episode outcome and mode labels plus timing logs, compute success rate, mean time-to-success, and per-mode counts with 95% Wilson intervals using a script; write output/metrics.json.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Python · pandas
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Control pair of T02: all metrics compared to grader computation (exact counts, rates within 1e-6, interval bounds within 1e-4).
Proposed partial credit
40% counts; 30% rates; 30% intervals.
R07.09.02.02-T15 Timestamp a gripper slip in a real-robot recording
Original instruction
Open fixtures/ur5_slip.mcap in Foxglove Studio (wrist camera + gripper width + FT sensor). Find the moment the grasped part begins to slip and write the timestamp (s, 0.1 s resolution) to output/answer.txt.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Foxglove Studio
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollav· Audio/video over timevis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader reference from expert consensus aligned with FT discontinuity: within +/-0.2 s.
Proposed partial credit
50% timestamp within 0.5 s; 50% timestamp within 0.2 s.
R07.09.02.02-T16 Check whether language-conditioned rollouts followed the instruction
Original instruction
fixtures/lang_eval/ contains 30 rollouts with instruction text (e.g., 'put the red block in the blue bowl'). Label each as followed, wrong object, wrong target or incomplete, in output/instruction_labels.csv.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- VLC · LeRobot
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader compares to labels from privileged object/target logs; accuracy over 30. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
Proposed partial credit
50% episodes 1-15 labeled correctly; 50% episodes 16-30 labeled correctly (pro-rata).
R07.09.02.02-T17 Write a failure-analysis brief with timestamped clips for the policy team
Original instruction
From the 80 labeled failures in fixtures/drawer_eval/ (videos, labels you produce), write output/brief.md naming the top 3 failure modes by frequency, 3 example episodes with timestamps each, trimmed clips in output/clips/ (ffmpeg, +/-2 s around the event) and a data-collection recommendation per mode.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- VLC · ffmpeg · Markdown editor
- Interfaces
desk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
av· Audio/video over timevis· Visual judgmentcode· Code & commandsmulti· Cross-app- Work product
WP01· Document- Conditions
long· Long-horizonmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Grader: top-3 modes match privileged-log ranking; each cited example is a true instance (time +/-0.5 s); clips contain the event; recommendations scored by a rubric LLM judge (only part not deterministic).
Proposed partial credit
40% top-3 modes; 30% examples and clips valid; 30% recommendations (rubric).
R07.09.02.02-T18 Mark the point of no return before each failure for DAgger review
Original instruction
For each of the 20 failed rollouts in fixtures/dagger_review/, mark the earliest frame at which a human should have intervened (after which failure is unavoidable under input/criteria.md). Use the provided review web app and export output/interventions.json.
- Platforms
web· Any OS (web)- Applications
- teleop UIs · LeRobot
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader reference is computed from counterfactual sim replays (from each frame, does the expert policy recover?): frame within +/-0.3 s. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
Proposed partial credit
50% fraction of episodes marked within 1 s; 50% fraction marked within 0.3 s.
R07.09.02.02-T19 Find mislabeled outcomes in a crowd-labeled rollout set
Original instruction
fixtures/crowd_labels.csv holds crowd success/fail labels for 40 videos in fixtures/stack_eval/. Review the videos and list the episode ids whose label is wrong in output/mislabeled.txt.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader has 8 planted label errors; F1 over episode ids. Sim success flags are withheld and the agent VM has no terminal or Python, so videos can only be reviewed in VLC.
Proposed partial credit
50% recall of the 8 planted label errors; 50% precision of listed ids.
R07.09.02.02-T20 Detect camera desynchronization across multi-view episodes
Original instruction
For the 20 episodes in fixtures/multiview/ (top, wrist, side cameras), use the gripper-close event visible in each view to estimate inter-camera offsets and list episodes where any pair is offset by more than 2 frames, with the offsets, in output/sync.csv.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- VLC · ffmpeg · Python
- Interfaces
desk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
av· Audio/video over timevis· Visual judgmentcode· Code & commands- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader: 5 planted desync episodes with known offsets; F1 over episodes; offsets within +/-1 frame.
Proposed partial credit
60% episode F1; 40% offsets within 1 frame.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.