策略回放审查与失败标注
Policy rollout review & failure labeling
Review robot policy rollouts and classify failures with timestamps.
20 条本项目编写的提案 · 分类 v1.1 · 新增基准运行 0 次。下方验证方式与部分得分规则是设计，不是执行结果。
- 典型应用
- LeRobot · Isaac Lab · MuJoCo viewer
- 小类 GUI 需求标签
HY· 混合
任务提案 · 20
展开查看完整原文与建议检查方式；两种界面均保留提案英文原文。
R07.09.02.02-T01 Label success or failure for ten pick-and-place rollout videos
原始指令
Watch the 10 rollout videos in fixtures/rollouts_pick/ (Franka, 'put the mug on the coaster') and write output/labels.csv with episode_id and outcome (success/failure). Success means the mug rests upright on the coaster at the final frame.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- MuJoCo viewer · VLC
- 交互表面
desk· 桌面 GUI- GUI 需求 · 任务设定
G2· GUI 必要- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 2 / 5
建议验证方式
Grader compares to ground truth from privileged sim logs; accuracy over 10 episodes. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
建议部分得分
50% episodes 1-5 correctly labeled; 50% episodes 6-10 correctly labeled (10 pts each).
R07.09.02.02-T02 Classify failure modes and onset times for 30 failed episodes
原始指令
For the 30 failed episodes in fixtures/fail_set_A/ (videos + joint logs), label each with one mode from input/taxonomy.md (missed grasp, slip in transport, collision, wrong object, timeout) and the onset time (s) when the failure became inevitable. Write output/failures.csv.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- LeRobot · VLC · Python
- 交互表面
desk· 桌面 GUIterm· 终端- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断code· 代码与命令- 工作产物
WP10· 标注- 条件
long· 长流程- 提案难度
- 3 / 5
建议验证方式
Grader compares to labels derived from privileged contact/grasp logs: mode accuracy and onset within +/-0.5 s. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
建议部分得分
60% mode accuracy (pro-rata); 40% onset within tolerance (pro-rata).
R07.09.02.02-T03 Find low-quality teleop episodes with the LeRobot dataset visualizer
原始指令
Open the 50-episode dataset fixtures/aloha_insert_v2 in the LeRobot dataset visualizer. Find episodes with frozen camera streams (>=1 s identical frames), action spikes (jumps >0.3 rad in one step) or missing gripper motion, and list them with the issue type in output/bad_episodes.csv.
- 平台
lin· Linuxmac· macOS- 应用
- LeRobot
- 交互表面
brow· 浏览器term· 终端- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
av· 音视频时序vis· 视觉判断nav· 平移 / 缩放 / 滚动- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader has planted issues in 7 episodes; F1 over (episode, issue_type) pairs.
建议部分得分
50% recall of planted (episode, issue) pairs; 50% precision of reported pairs.
R07.09.02.02-T04 Evaluate an Isaac Lab checkpoint over 100 seeds and label failures
原始指令
Run fixtures/checkpoints/lift_cube_v3.pt in Isaac Lab's Franka lift task for seeds 0-99 with video recording, report the success rate, and label each failed seed with a mode (no grasp, drop, overshoot, timeout) in output/eval.csv. Use the provided play script; do not retrain.
- 平台
lin· Linux- 应用
- Isaac Lab
- 交互表面
term· 终端desk· 桌面 GUI- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令av· 音视频时序vis· 视觉判断- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Grader reruns the deterministic evaluation: success rate exact; failure-mode labels compared to privileged per-seed event logs (accuracy).
建议部分得分
30% evaluation runs with correct seeds; 20% success rate exact; 50% failure mode accuracy.
R07.09.02.02-T05 Locate the fall and first contact loss in a humanoid recording
原始指令
Open fixtures/humanoid_walk.rrd in the Rerun viewer. Find the time the humanoid falls (pelvis height <0.4 m) and which foot first lost stable contact before the fall, and record both in output/answer.json (time in s, foot = left/right).
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- Rerun
- 交互表面
desk· 桌面 GUI- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
nav· 平移 / 缩放 / 滚动vis· 视觉判断av· 音视频时序- 工作产物
WP12· 答案 / 决策- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader computes from the recording's hidden ground-truth channels: fall time within 0.1 s, foot exact.
建议部分得分
50% fall time; 50% foot.
R07.09.02.02-T06 Compare two policies seed by seed and explain B's regressions
原始指令
Policies A and B were run on the same 40 seeds (fixtures/ab_eval/{A,B}/ videos and logs). List seeds where A succeeds and B fails, and label B's failure mode per input/taxonomy.md with onset time. Write output/regressions.csv.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- LeRobot · Python · VLC
- 交互表面
desk· 桌面 GUIterm· 终端- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断code· 代码与命令- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Grader compares the regression seed set (F1) and modes/onsets against privileged logs (onset +/-0.5 s). Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
建议部分得分
40% regression set F1; 40% mode accuracy; 20% onsets.
R07.09.02.02-T07 Report the step where an object first hits the floor
原始指令
In MuJoCo simulate, load fixtures/shelf_drop.xml and play the recorded trajectory fixtures/traj_17.mjtraj; report the simulation step at which the box first touches the floor in output/answer.txt.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- MuJoCo viewer
- 交互表面
desk· 桌面 GUI- GUI 需求 · 任务设定
G2· GUI 必要- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
nav· 平移 / 缩放 / 滚动vis· 视觉判断- 工作产物
WP12· 答案 / 决策- 条件
- 未声明
- 提案难度
- 2 / 5
建议验证方式
Grader computes first floor contact from the trajectory; step within +/-2. The agent VM has no terminal or MuJoCo Python bindings and contact data is withheld, so the step must be found by scrubbing playback in simulate.
建议部分得分
50% step within 10 of truth; 50% step within 2 of truth.
R07.09.02.02-T08 Annotate failure spans across 200 rollouts and build a mode histogram
原始指令
Import the 200 rollout videos in fixtures/bin_pick_200/ into the provided CVAT project, annotate a temporal span for every failure event with its mode label, export the annotations, and write output/histogram.csv plus one representative clip per mode (output/clips/<mode>.mp4).
- 平台
web· 任意系统（网页）- 应用
- CVAT · ffmpeg
- 交互表面
brow· 浏览器term· 终端- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
HY· 混合- 交互原语
av· 音视频时序vis· 视觉判断drag· 拖放- 工作产物
WP10· 标注- 条件
long· 长流程multiapp· 多应用- 提案难度
- 5 / 5
建议验证方式
Grader matches exported spans to privileged-event spans: temporal IoU >=0.5 counts as match; label accuracy on matches; histogram from labels must equal counts; each clip must contain a true event of its mode.
建议部分得分
40% span F1; 30% label accuracy; 15% histogram; 15% clips valid.
R07.09.02.02-T09 Segment subtask phases in fifteen demonstration videos
原始指令
In Label Studio (video timeline), segment each of the 15 videos in fixtures/demos/ into phases reach, grasp, lift, transport and place, following input/phase_rules.md, and export JSON to output/phases.json.
- 平台
web· 任意系统（网页）- 应用
- Label Studio
- 交互表面
brow· 浏览器- GUI 需求 · 任务设定
G2· GUI 必要- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序drag· 拖放vis· 视觉判断- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader compares to reference segmentation from gripper/object state logs: mean per-phase temporal IoU; full credit at >=0.7. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
建议部分得分
50% for mean temporal IoU up to 0.35 (pro-rata); 50% for mean temporal IoU from 0.35 to 0.7 (pro-rata).
R07.09.02.02-T10 Flag rollouts with self-collisions or joint-limit violations
原始指令
Review the 60 rollouts in fixtures/bimanual_eval/ (videos + joint_state CSVs + URDF limits) and list every episode with a self-collision or joint-limit violation, its time, the joints/links involved and severity per input/severity.md in output/safety.csv.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- Isaac Lab · Python · VLC
- 交互表面
desk· 桌面 GUIterm· 终端- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令av· 音视频时序vis· 视觉判断- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Grader: joint-limit events derivable from CSV (exact match required); self-collisions from hidden contact logs (visual-only for agent): F1 over episodes, time +/-0.3 s, severity accuracy.
建议部分得分
35% joint-limit events; 35% self-collision events; 15% times; 15% severity.
R07.09.02.02-T11 Find lucky successes where the object reached the goal by accident
原始指令
All 25 episodes in fixtures/push_success/ are flagged success by the env. Watch them and mark episodes where the goal was reached by bounce, knock-over or after leaving the table, rather than a controlled push, in output/lucky.csv with the time of the uncontrolled event.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- MuJoCo viewer · VLC
- 交互表面
desk· 桌面 GUI- GUI 需求 · 任务设定
G2· GUI 必要- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Grader has 6 planted lucky episodes (from contact logs); F1 over episodes and event time within 0.5 s. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
建议部分得分
70% episode F1; 30% event times.
R07.09.02.02-T12 Mark where real-robot runs diverge from their sim twins
原始指令
fixtures/sim2real/ holds 12 pairs of sim and real videos of the same policy and initial state. For each real episode, label whether it diverges from sim and, if so, the phase (approach, grasp, lift, place) and time where divergence starts, in output/divergence.csv.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- VLC · LeRobot
- 交互表面
desk· 桌面 GUI- GUI 需求 · 任务设定
G2· GUI 必要- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader compares to expert reference labels (two-annotator consensus): divergence flag accuracy, phase accuracy, time within 1 s. Sim state and robot logs are withheld from the agent; only the video pairs are provided, so divergence must be judged by viewing them.
建议部分得分
40% divergence flags; 35% phase; 25% time.
R07.09.02.02-T13 Spot reward-hacking behavior in training-time rollouts
原始指令
Rollouts from an Isaac Lab peg-insert run (fixtures/peg_hack/, 40 videos + reward logs) show rising reward. Identify the episodes where reward is earned by an exploit (e.g., peg clipping through the board or tilting the socket) and describe the exploit in output/hack_report.md with episode ids and times.
- 平台
lin· Linux- 应用
- Isaac Lab · VLC · Python
- 交互表面
desk· 桌面 GUIterm· 终端- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断code· 代码与命令- 工作产物
WP01· 文档- 条件
- 未声明
- 提案难度
- 5 / 5
建议验证方式
Grader: planted exploit episodes (from physics penetration logs) F1 >=0.8 for full credit; exploit type must match one of 2 planted mechanisms (keyword rubric); times within 0.5 s. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
建议部分得分
50% episode F1; 30% exploit mechanisms named; 20% times.
R07.09.02.02-T14 Compute rollout metrics from labeled logs (control pair of T02)
原始指令
Control pair of T02: given fixtures/fail_set_A/labels_gt.parquet with per-episode outcome and mode labels plus timing logs, compute success rate, mean time-to-success, and per-mode counts with 95% Wilson intervals using a script; write output/metrics.json.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- Python · pandas
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP02· 记录 / 表格- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Control pair of T02: all metrics compared to grader computation (exact counts, rates within 1e-6, interval bounds within 1e-4).
建议部分得分
40% counts; 30% rates; 30% intervals.
R07.09.02.02-T15 Timestamp a gripper slip in a real-robot recording
原始指令
Open fixtures/ur5_slip.mcap in Foxglove Studio (wrist camera + gripper width + FT sensor). Find the moment the grasped part begins to slip and write the timestamp (s, 0.1 s resolution) to output/answer.txt.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- Foxglove Studio
- 交互表面
desk· 桌面 GUI- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
nav· 平移 / 缩放 / 滚动av· 音视频时序vis· 视觉判断- 工作产物
WP12· 答案 / 决策- 条件
- 未声明
- 提案难度
- 2 / 5
建议验证方式
Grader reference from expert consensus aligned with FT discontinuity: within +/-0.2 s.
建议部分得分
50% timestamp within 0.5 s; 50% timestamp within 0.2 s.
R07.09.02.02-T16 Check whether language-conditioned rollouts followed the instruction
原始指令
fixtures/lang_eval/ contains 30 rollouts with instruction text (e.g., 'put the red block in the blue bowl'). Label each as followed, wrong object, wrong target or incomplete, in output/instruction_labels.csv.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- VLC · LeRobot
- 交互表面
desk· 桌面 GUI- GUI 需求 · 任务设定
G2· GUI 必要- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader compares to labels from privileged object/target logs; accuracy over 30. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
建议部分得分
50% episodes 1-15 labeled correctly; 50% episodes 16-30 labeled correctly (pro-rata).
R07.09.02.02-T17 Write a failure-analysis brief with timestamped clips for the policy team
原始指令
From the 80 labeled failures in fixtures/drawer_eval/ (videos, labels you produce), write output/brief.md naming the top 3 failure modes by frequency, 3 example episodes with timestamps each, trimmed clips in output/clips/ (ffmpeg, +/-2 s around the event) and a data-collection recommendation per mode.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- VLC · ffmpeg · Markdown editor
- 交互表面
desk· 桌面 GUIterm· 终端- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
HY· 混合- 交互原语
av· 音视频时序vis· 视觉判断code· 代码与命令multi· 跨应用- 工作产物
WP01· 文档- 条件
long· 长流程multiapp· 多应用- 提案难度
- 5 / 5
建议验证方式
Grader: top-3 modes match privileged-log ranking; each cited example is a true instance (time +/-0.5 s); clips contain the event; recommendations scored by a rubric LLM judge (only part not deterministic).
建议部分得分
40% top-3 modes; 30% examples and clips valid; 30% recommendations (rubric).
R07.09.02.02-T18 Mark the point of no return before each failure for DAgger review
原始指令
For each of the 20 failed rollouts in fixtures/dagger_review/, mark the earliest frame at which a human should have intervened (after which failure is unavoidable under input/criteria.md). Use the provided review web app and export output/interventions.json.
- 平台
web· 任意系统（网页）- 应用
- teleop UIs · LeRobot
- 交互表面
brow· 浏览器- GUI 需求 · 任务设定
G2· GUI 必要- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断nav· 平移 / 缩放 / 滚动- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Grader reference is computed from counterfactual sim replays (from each frame, does the expert policy recover?): frame within +/-0.3 s. Privileged sim logs (contacts, success flags) are withheld from the agent; only videos/recordings are provided.
建议部分得分
50% fraction of episodes marked within 1 s; 50% fraction marked within 0.3 s.
R07.09.02.02-T19 Find mislabeled outcomes in a crowd-labeled rollout set
原始指令
fixtures/crowd_labels.csv holds crowd success/fail labels for 40 videos in fixtures/stack_eval/. Review the videos and list the episode ids whose label is wrong in output/mislabeled.txt.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- VLC
- 交互表面
desk· 桌面 GUI- GUI 需求 · 任务设定
G2· GUI 必要- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
av· 音视频时序vis· 视觉判断- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader has 8 planted label errors; F1 over episode ids. Sim success flags are withheld and the agent VM has no terminal or Python, so videos can only be reviewed in VLC.
建议部分得分
50% recall of the 8 planted label errors; 50% precision of listed ids.
R07.09.02.02-T20 Detect camera desynchronization across multi-view episodes
原始指令
For the 20 episodes in fixtures/multiview/ (top, wrist, side cameras), use the gripper-close event visible in each view to estimate inter-camera offsets and list episodes where any pair is offset by more than 2 frames, with the offsets, in output/sync.csv.
- 平台
mac· macOSwin· Windowslin· Linux- 应用
- VLC · ffmpeg · Python
- 交互表面
desk· 桌面 GUIterm· 终端- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
HY· 混合- 交互原语
av· 音视频时序vis· 视觉判断code· 代码与命令- 工作产物
WP10· 标注- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader: 5 planted desync episodes with known offsets; F1 over episodes; offsets within +/-1 frame.
建议部分得分
60% episode F1; 40% offsets within 1 frame.
已有研究映射
以下沿用研究目录中的基准与小类映射，不是逐题标注、执行记录或产品背书。