Teleoperated assembly & dexterous manipulation
Teleoperate arms and dexterous hands in simulation to assemble, insert and move objects.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- MuJoCo · Isaac Sim · LeRobot · teleop UIs
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R07.09.02.01-T01 Push a cube into a target square with MuJoCo mouse perturbation
Original instruction
In MuJoCo simulate, load fixtures/push_table.xml (a 5 cm cube on a table with a 10 cm target square). Using only Ctrl+right-drag force perturbation on the cube while the sim runs in real time, push it so its center rests inside the square with yaw error <15 deg.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader reads the final state from simulate's recorded trajectory: cube center inside target, yaw error <15 deg, cube never left the table. Only perturbation events in the log may move the cube; qpos edits via the UI panel are detected and fail.
Proposed partial credit
50% cube center inside square (half of that if within 5 cm); 30% yaw; 20% stayed on table.
R07.09.02.01-T02 Pick a block and place it on a coaster with keyboard teleop
Original instruction
In the browser teleop UI for the simulated Franka (fixtures/teleop_pick), use the keyboard end-effector jog controls (translate/rotate, gripper open/close) to pick the red 4 cm block and place it upright on the coaster, centered within 1 cm.
- Platforms
web· Any OS (web)- Applications
- teleop UIs · MuJoCo
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsrt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader reads the episode log: block on coaster with center error <1 cm and tilt <5 deg after release, no block drop >5 cm. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
30% grasp achieved; 50% placement within 1 cm (half of that within 3 cm); 20% upright and no drops.
R07.09.02.01-T03 Insert a peg into a 0.5 mm-clearance hole under a force limit
Original instruction
In Isaac Sim with the 6-DoF space-mouse emulator panel controlling the UR5e (fixtures/peg_hole.usd), insert the 20 mm peg into the hole with 0.5 mm clearance to a depth of at least 20 mm while keeping wrist force below 30 N throughout.
- Platforms
lin· Linuxwin· Windows- Applications
- Isaac Sim · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
drag· Drag & droprt· Real-time controlgeom· Precise geometryvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader reads the logged contact/FT stream: final insertion depth >=20 mm, max |F| <30 N, no penetration beyond 1 mm into the hole wall (physics state). Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
20% peg aligned over hole; 50% depth reached (pro-rata); 30% force limit respected.
R07.09.02.01-T04 Reorient a cube in-hand with a dexterous hand to show a target face
Original instruction
In MuJoCo, teleoperate the Shadow Hand (fixtures/shadow_cube.xml) through the provided joint-synergy sliders so the lettered cube shows face 'C' upward with orientation error below 15 deg, without dropping the cube or touching the table.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader computes the cube quaternion at end: face C normal within 15 deg of +Z, cube never contacted the table and remained in the hand (palm-distance check). Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
50% orientation (pro-rata to 45 deg); 30% no drop; 20% no table contact.
R07.09.02.01-T05 Hand a cube from one ALOHA arm to the other
Original instruction
In LeRobot's gym_aloha transfer-cube task, use the browser leader-arm teleop UI to pick the cube with the right arm, hand it over mid-air to the left arm and hold it above the table for 2 s. Record the episode to output/episode_0.
- Platforms
lin· Linuxmac· macOS- Applications
- LeRobot · MuJoCo · teleop UIs
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader replays the recorded episode in the env: task success flag true, handover happens with cube height >5 cm, hold >=2 s, episode saved in LeRobot dataset format. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
25% right-arm grasp; 35% handover; 25% hold 2 s; 15% episode saved.
R07.09.02.01-T06 Assemble three gears onto shafts on a simulated task board
Original instruction
In fixtures/nist_taskboard.usd (Isaac Sim), teleoperate the Franka with the space-mouse panel to place the small, medium and large gears onto their shafts in that order, each fully seated and meshed, within 15 minutes of sim time.
- Platforms
lin· Linuxwin· Windows- Applications
- Isaac Sim · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlgeom· Precise geometryvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader reads final gear poses: each concentric with its shaft within 0.5 mm and seated within 1 mm of the stop; order from event log; teeth mesh check (relative angle) for adjacent gears. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
75% gears seated (25 pts each); 15% correct order; 10% meshing.
R07.09.02.01-T07 Stack three blocks into a tower that stays up for five seconds
Original instruction
In the MuJoCo teleop UI (fixtures/stack3), use the end-effector jog controls to stack the blue, green and red blocks (bottom to top) on the marked spot so the tower stands for 5 s after release with top-block offset <1 cm.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
key· Shortcutsdrag· Drag & droprt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader checks final state 5 s after last release: order blue-green-red, each block's xy offset <1 cm from the one below, all velocities ~0. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
30% two-block stack; 40% three-block stack in order; 30% stable 5 s with offsets in tolerance.
R07.09.02.01-T08 Route a deformable cable through three clips
Original instruction
In fixtures/cable_route.xml (MuJoCo, flexible cable), teleoperate the two grippers to route the cable from its anchor through clips C1, C2 and C3 in order and leave it seated in all three, with the free end past C3.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlpath· Continuous pathvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader checks cable body segments: for each clip at least one segment inside the clip slot volume, route order along cable arc length C1<C2<C3, cable not snagged on other geometry. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
75% clips correctly seated (25 pts each); 15% correct order; 10% free end past C3.
R07.09.02.01-T09 Open a drawer, place a mug inside and close it
Original instruction
In the kitchen scene fixtures/drawer_mug.usd (Isaac Sim), teleoperate the Franka to open the top drawer at least 20 cm, place the mug upright inside it, then close the drawer to within 1 cm of fully closed.
- Platforms
lin· Linuxwin· Windows- Applications
- Isaac Sim · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader reads final joint of the drawer and mug pose: mug inside drawer volume and upright (<10 deg), drawer joint <1 cm; event log shows drawer opened >=20 cm. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
30% drawer opened; 40% mug placed upright inside; 30% drawer closed.
R07.09.02.01-T10 Pour beads from a cup into a bowl with minimal spill
Original instruction
In fixtures/pour.usd (Isaac Sim particles), teleoperate the arm holding a cup of 500 beads so that at least 80% of beads end up in the bowl and fewer than 5% fall on the table, without the cup touching the bowl.
- Platforms
lin· Linuxwin· Windows- Applications
- Isaac Sim · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader counts particle positions at the end: fraction in bowl volume, fraction on table; contact log for cup-bowl contact. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
60% beads-in-bowl fraction (pro-rata from 0.4 to 0.8); 25% spill under 5 pct; 15% no cup-bowl contact.
R07.09.02.01-T11 Place a block on target under 300 ms teleop latency
Original instruction
The teleop UI for fixtures/latency_place adds 300 ms command and video latency. Pick the cylinder and place it on the target ring within 1 cm, without knocking over the two nearby distractor cylinders.
- Platforms
web· Any OS (web)- Applications
- teleop UIs · MuJoCo
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsrt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader: final cylinder center within 1 cm of target and upright; distractors upright (tilt <10 deg); latency setting verified unchanged in the session config. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
50% placement; 30% distractors untouched; 20% completed under 4 min sim time.
R07.09.02.01-T12 Record five successful push-T teleop demonstrations in LeRobot
Original instruction
Run LeRobot's gym_pusht teleop recorder (mouse-controlled pusher) and record 5 episodes into output/pusht_demos in LeRobot dataset format. At least 4 must reach coverage >=0.95 of the T target within 300 steps.
- Platforms
lin· Linuxmac· macOS- Applications
- LeRobot · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlvis· Visual judgment- Work product
WP05· Time-based media- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader loads the dataset with LeRobot: 5 episodes, valid schema and video frames; replays actions in the env and counts episodes with coverage >=0.95 within 300 steps. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
20% dataset valid; 80% successful episodes (20 pts each, up to 4).
R07.09.02.01-T13 Thread a nut onto a bolt with wrist rotation
Original instruction
In fixtures/nut_bolt.xml (MuJoCo), teleoperate the 7-DoF arm and gripper to pick the M12 nut and thread it onto the vertical bolt at least 3 full turns using regrasp-and-rotate cycles.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlgeom· Precise geometry- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader reads the nut's screw-joint angle: >=1080 deg accumulated, axial position consistent with thread pitch; no cross-thread (nut tilt <3 deg). Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
20% nut started on thread; 60% turns (pro-rata to 3); 20% no cross-thread.
R07.09.02.01-T14 Drive a mobile manipulator to a shelf and pick a box
Original instruction
In fixtures/shelf_store.usd (Isaac Sim), use the teleop UI (WASD base, arm jog panel) to drive the mobile manipulator through the aisle without collisions, pick the box labeled 'SKU-417' from shelf level 2 and bring it to the drop zone.
- Platforms
lin· Linuxwin· Windows- Applications
- Isaac Sim · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsdrag· Drag & droprt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader: correct box (prim id) inside drop zone at end; base-shelf collisions = 0 in contact log; other boxes on the shelf displaced <2 cm. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
20% reached shelf without collisions; 40% correct box picked; 30% delivered; 10% other boxes undisturbed.
R07.09.02.01-T15 Press four panel buttons in a given sequence
Original instruction
In the teleop UI for fixtures/button_panel, press the buttons in the order Green, Blue, Blue, Red using the robot fingertip; each press must exceed 5 mm travel and no other button may be pressed.
- Platforms
web· Any OS (web)- Applications
- teleop UIs · MuJoCo
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsrt· Real-time control- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader reads the button joint log: exact press sequence with travel >5 mm each, no extra presses. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
25% first press correct; 25% second; 25% third; 25% fourth (a press scores 0 if an extra press intervenes).
R07.09.02.01-T16 Fold a cloth in half with aligned corners
Original instruction
In fixtures/cloth_fold.xml (MuJoCo flex cloth, 40x40 cm), teleoperate the two grippers to fold the cloth in half along its vertical midline so that the two left corners lie on the two right corners within 3 cm, and leave it flat.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- MuJoCo · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlpath· Continuous pathvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader reads final vertex positions: corner pair distances <3 cm, folded area 45-55% of original footprint, max height <4 cm. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
40% corner alignment; 40% folded area; 20% flatness.
R07.09.02.01-T17 Insert a USB-C plug with 0.2 mm tolerance
Original instruction
In fixtures/usbc_insert.usd (Isaac Sim), teleoperate the gripper holding a USB-C plug (cable attached) into the laptop port: align orientation, insert to full depth with clearance 0.2 mm, peak force <15 N, without bending the cable over 90 deg.
- Platforms
lin· Linuxwin· Windows- Applications
- Isaac Sim · teleop UIs
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlgeom· Precise geometryvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Grader: plug depth within 0.5 mm of fully seated, axis misalignment <1 deg at end, max contact force <15 N, cable bend <90 deg (segment angles). Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
20% aligned at port; 50% fully inserted; 20% force limit; 10% cable bend.
R07.09.02.01-T18 Script an impedance-control peg insertion (control pair of T03)
Original instruction
Control pair of T03: without the teleop UI, write output/insert_peg.py using the Isaac Sim Python API with a Cartesian impedance or spiral-search strategy to insert the same 20 mm peg (0.5 mm clearance) to >=20 mm depth with force <30 N.
- Platforms
lin· Linuxwin· Windows- Applications
- Isaac Sim · Python
- Interfaces
api· API/SDKterm· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Control pair of T03: grader runs the script on 5 seeds with +/-2 mm initial offsets: success rate, depth and force thresholds as in T03.
Proposed partial credit
20% script runs; 60% success rate over 5 seeds; 20% force limit in all runs.
R07.09.02.01-T19 Hand over a tool between two arms in a WebXR teleop session
Original instruction
In the Isaac Sim WebXR teleop session (WebXR emulator in the browser, hand-tracking poses driven by the emulator controls), grasp the screwdriver with the left robot hand, hand it to the right hand and place it in the tool tray slot.
- Platforms
web· Any OS (web)- Applications
- Isaac Sim · teleop UIs
- Interfaces
xr· XR/spatialbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droprt· Real-time controlgeom· Precise geometryvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader: event log shows left grasp, simultaneous two-hand contact, right-hand-only hold; final screwdriver pose in slot within 1 cm/10 deg; no drop. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
25% left grasp; 35% handover; 30% slotted within tolerance; 10% no drop.
R07.09.02.01-T20 Recover after a scripted disturbance knocks the part loose
Original instruction
In the teleop UI for fixtures/kit_assembly, pick the bracket and place it in fixture slot 2. Partway through, a scripted disturbance knocks the bracket out of the gripper; recover, re-grasp and complete the placement within the 6-minute session.
- Platforms
web· Any OS (web)- Applications
- teleop UIs · MuJoCo
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsrt· Real-time controlvis· Visual judgment- Work product
WP11· Game/sim state- Conditions
interrupt· Interruptions- Proposed difficulty
- 3 / 5
Proposed verifier
Grader: disturbance event fired; final bracket pose in slot 2 within 5 mm/5 deg; completion time <6 min; other kit parts unmoved. Sim state/control APIs are not exposed to the agent; grader checks that every command in the control log came from the teleop UI process.
Proposed partial credit
20% initial grasp; 30% re-grasp after disturbance; 40% final placement; 10% other parts untouched.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.