R13.04.01.01 · TASK CLASS

Flight & driving simulators

Complete procedures and maneuvers in sims to tolerance.

20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.

Typical applications: X-Plane · MSFS · CARLA
Class GUI-need label: G3 · GUI-only

Task proposals · 20

Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.

R13.04.01.01-T01 Start a Cessna 172 from cold and dark in FlightGear

Original instruction

FlightGear (desktop) loads the c172p at KPAO parking, cold and dark. Following input/c172_startup_checklist.pdf, set fuel selector BOTH, mixture rich, master and beacon on, prime, magnetos to START and run the engine at 1,000 RPM with avionics on. Use the cockpit controls with the mouse.

Platforms: mac · macOS win · Windows lin · Linux
Applications: FlightGear
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: click · Point & click drag · Drag & drop vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: tutorial · Follow docs/tutorial
Proposed difficulty: 3 / 5

Proposed verifier

Property tree logged via the harness's telnet port (not reachable from the agent's VM): engine running, rpm 900-1,100, magnetos BOTH, mixture ≥ 0.9, master/beacon/avionics on, fuel selector BOTH. The in-sim property browser and Nasal console are disabled.

Proposed partial credit

40% switch states per checklist; 20% priming/mixture; 40% engine running at target RPM

R13.04.01.01-T02 Take off and hold heading, altitude and speed in X-Plane

Original instruction

X-Plane 12 Cessna 172 lined up on KSEA runway 34R, clear weather. Take off, climb to 3,000 ft MSL, then hold heading 340° ±10°, altitude 3,000 ±100 ft and 90 ±5 KIAS for 60 continuous seconds using keyboard/mouse flight controls (autopilot off).

Platforms: mac · macOS win · Windows lin · Linux
Applications: X-Plane
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Harness reads datarefs via a UDP plugin bound to the host only: autopilot mode off throughout; longest window meeting all tolerances ≥ 60 s. Agent sees the rendered cockpit only. X-Plane's UDP port is firewalled from the agent VM and no FlyWithLua/XPPython3 plugins are installed, so datarefs cannot be written by script.

Proposed partial credit

20% takeoff; 20% reach 3,000 ft; 60% scaled by longest in-tolerance window up to 60 s

R13.04.01.01-T03 Land on a runway within the touchdown zone in Microsoft Flight Simulator

Original instruction

MSFS 2020/2024, Cessna 172 on a 5 nm final to KPAO runway 31 at 1,500 ft, clear weather. Land with touchdown in the first third of the runway, vertical speed at touchdown ≤ 300 fpm, centreline deviation ≤ 5 m, then stop on the runway.

Platforms: win · Windows
Applications: MSFS
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 5 / 5

Proposed verifier

SimConnect logger (harness side) records touchdown position, vertical speed and lateral offset; aircraft stopped on runway, no crash flag. Assistance options (auto-rudder, landing assist) are disabled.

Proposed partial credit

30% safe landing (no crash, stopped on runway); 25% touchdown zone; 25% vertical speed; 20% centreline

R13.04.01.01-T04 Program a G1000 flight plan and engage NAV mode in X-Plane

Original instruction

X-Plane 12 Cessna 172 G1000 at KPAO, engine running. Using the G1000's FPL page knobs, enter KPAO → OSI → SNS → KSBA, activate the plan, take off and engage the autopilot in NAV and ALT mode at 7,500 ft so the aircraft tracks toward OSI.

Platforms: mac · macOS win · Windows lin · Linux
Applications: X-Plane
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click drag · Drag & drop vis · Visual judgment
Work product: WP08 · App/system state
Conditions: tutorial · Follow docs/tutorial
Proposed difficulty: 4 / 5

Proposed verifier

Datarefs via harness plugin: FMS entries equal the 4 waypoints in order; AP mode NAV and ALT, altitude target 7,500; cross-track error to the active leg < 0.3 nm for 2 minutes. X-Plane's UDP port is firewalled from the agent VM and no FlyWithLua/XPPython3 plugins are installed, so the plan cannot be loaded by script.

Proposed partial credit

40% flight plan exact; 20% AP modes; 20% altitude target; 20% tracking

R13.04.01.01-T05 Fly a full traffic pattern and land in FlightGear

Original instruction

FlightGear c172p on KPAO runway 31, engine running. Fly a left-hand traffic pattern at 1,000 ft AGL ±100 (crosswind, downwind, base, final) and land on runway 31, rolling to a stop on the runway.

Platforms: mac · macOS win · Windows lin · Linux
Applications: FlightGear
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: long · Long-horizon
Proposed difficulty: 4 / 5

Proposed verifier

Position/altitude trace from the property tree (harness): pattern legs inside a geometric template; downwind altitude within band; touchdown on runway 31 with vertical speed < 600 fpm; final stop on the runway. The telnet/HTTP property server is not reachable from the agent's VM, and the in-sim property browser and Nasal console are disabled.

Proposed partial credit

30% pattern geometry; 20% altitude discipline; 50% landing and stop

R13.04.01.01-T06 Drive a CARLA route obeying traffic lights without collisions

Original instruction

CARLA 0.9.15 Town03 with a manual-control window (keyboard WASD). Drive from spawn point 12 to the marked destination (route shown on the HUD minimap), stopping at red lights, with no collisions and finishing within 5 minutes.

Platforms: lin · Linux win · Windows
Applications: CARLA
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

CARLA ScenarioRunner-style criteria computed from the server recording: route completion %, red-light infractions, collisions, time. The agent's client is the GUI window only; Python client API port is firewalled from the agent VM.

Proposed partial credit

50% scaled by route completion; 20% no red-light runs; 20% no collisions; 10% under 5 min

R13.04.01.01-T07 Parallel park within tolerance in CARLA

Original instruction

CARLA Town01, manual control. A parking space between two parked cars is marked by the HUD. Parallel park so the car ends inside the space with centre offset ≤ 0.3 m, heading error ≤ 5° and no contact.

Platforms: lin · Linux win · Windows
Applications: CARLA
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: rt · Real-time control geom · Precise geometry vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Recorder log: final ego transform vs bay; collision sensor events = 0. The CARLA Python client port is firewalled from the agent VM, so the ego vehicle cannot be moved or driven by script; control comes only from the manual-control window.

Proposed partial credit

30% inside the space; 40% scaled by offset; 15% heading; 15% no contact

R13.04.01.01-T08 Configure time, weather and a scheduled failure in X-Plane

Original instruction

In X-Plane 12, before flight, set the date to 2026-12-21 at 16:30 local, weather to manual with wind 270° at 15 kt, broken clouds at 1,500 ft AGL and visibility 5 sm, and schedule a vacuum pump failure at 2,000 ft AGL. Start the flight at KBOS in the Cessna 172.

Platforms: mac · macOS win · Windows lin · Linux
Applications: X-Plane
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G3 · GUI-only
Interaction primitives: click · Point & click type · Text & forms nav · Pan/zoom/scroll
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Datarefs after flight start (harness plugin): local time and date, wind layer, cloud layer type/base, visibility within tolerance; failure entry for vacuum pump with altitude trigger 2,000 AGL. X-Plane's UDP port is firewalled from the agent VM and no FlyWithLua/XPPython3 plugins are installed, so settings cannot be written by script.

Proposed partial credit

25% time/date; 35% weather values; 30% failure scheduled; 10% correct airport and aircraft

R13.04.01.01-T09 Perform a 360° steep turn holding altitude in Microsoft Flight Simulator

Original instruction

MSFS, Cessna 172 at 3,500 ft, 100 KIAS, straight and level, autopilot off. Perform a 360° steep turn to the left at 45° ±5° bank, holding altitude ±100 ft and airspeed ±10 kt, rolling out on the entry heading ±10°.

Platforms: win · Windows
Applications: MSFS
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

SimConnect trace: heading change 360° ±10°, bank within 40-50° for ≥ 80% of the turn, altitude and speed deviations within tolerance (ACS steep-turn standard). SimConnect is not exposed to the agent (only the harness client is accepted) and no add-ons or external tools are connected, so controls cannot be driven by script.

Proposed partial credit

30% bank maintained; 30% altitude held; 20% airspeed; 20% rollout heading

R13.04.01.01-T10 Fly a coupled ILS approach to decision height in FlightGear

Original instruction

FlightGear c172p with KAP140 autopilot, 12 nm from KSFO runway 28R at 4,000 ft. Tune NAV1 to the ILS frequency from input/ksfo_ils28r.pdf, set the OBS, arm APR on the autopilot to capture localizer and glideslope, and disconnect at 200 ft AGL.

Platforms: mac · macOS win · Windows lin · Linux
Applications: FlightGear
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click drag · Drag & drop vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: tutorial · Follow docs/tutorial
Proposed difficulty: 3 / 5

Proposed verifier

Property tree: NAV1 frequency exact; AP APR mode with localizer and glideslope captured; deviation < 1 dot at 500 ft; AP disconnected between 150-250 ft AGL. The telnet/HTTP property server is not reachable from the agent's VM, and the in-sim property browser and Nasal console are disabled.

Proposed partial credit

25% frequency and OBS; 35% LOC/GS captured; 25% deviations; 15% disconnect at DH

R13.04.01.01-T11 Handle an engine failure after takeoff in X-Plane

Original instruction

X-Plane 12, Cessna 172 departing KHAF runway 30. A random engine failure is scheduled between 400 and 1,200 ft AGL. After it occurs, establish best glide, and land on the runway, an open field, or the beach without crashing, with the aircraft stopped and occupants safe.

Platforms: mac · macOS win · Windows lin · Linux
Applications: X-Plane
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: interrupt · Interruptions
Proposed difficulty: 5 / 5

Proposed verifier

Datarefs: failure event logged; airspeed within 63-73 KIAS for ≥ 70% of the glide; touchdown with no crash flag and vertical speed < 700 fpm; aircraft stopped upright. X-Plane's UDP port is firewalled from the agent VM and no FlyWithLua/XPPython3 plugins are installed, so controls and datarefs cannot be driven by script.

Proposed partial credit

30% best-glide airspeed held; 20% heading within ±30° of a landable area; 50% survivable landing

R13.04.01.01-T12 Hover a Robinson R22 in a box for thirty seconds in X-Plane

Original instruction

X-Plane 12, Robinson R22 on a helipad, engine running. Lift off and hover at 5-10 ft AGL inside a 3 m radius of the pad centre for 30 continuous seconds, then land on the pad.

Platforms: mac · macOS win · Windows lin · Linux
Applications: X-Plane
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 5 / 5

Proposed verifier

Datarefs: longest window with height 5-10 ft AGL and horizontal distance ≤ 3 m; final touchdown on the pad with no damage flags. X-Plane's UDP port is firewalled from the agent VM and no FlyWithLua/XPPython3 plugins are installed, so position and controls cannot be set by script.

Proposed partial credit

70% scaled by longest valid hover up to 30 s; 30% landing on pad

R13.04.01.01-T13 Stop for a crossing pedestrian in a CARLA scenario

Original instruction

CARLA scenario (Town01): driving at the speed limit, a pedestrian steps out from behind a parked van about 25 m ahead. Drive the route manually and stop or avoid without collision, then continue to the route end.

Platforms: lin · Linux win · Windows
Applications: CARLA
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: AP · API-sufficient
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: interrupt · Interruptions
Proposed difficulty: 3 / 5

Proposed verifier

ScenarioRunner criteria from the recording: collision with pedestrian = 0; route completed; ego speed before the event ≥ 80% of the limit (no crawling). The CARLA Python client port is firewalled from the agent VM, so the ego vehicle cannot be driven by script; control comes only from the manual-control window.

Proposed partial credit

60% no collision; 20% speed criterion; 20% route completed

R13.04.01.01-T14 Stop a train at the platform marker on schedule in OpenBVE

Original instruction

OpenBVE (desktop) runs a commuter route with a 6-car train. Drive from the first station to the third, stopping each time with the train's front within ±2 m of the stop marker and departing/arriving within ±30 s of the timetable shown in the cab.

Platforms: mac · macOS win · Windows lin · Linux
Applications: OpenBVE
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

OpenBVE score log/plugin output: stop position error per station and arrival time deviation; no overspeed or emergency brake penalties. Only the harness's logging plugin is loaded and the agent has no shell, so the train cannot be driven or the log written by script; control comes from the cab.

Proposed partial credit

40% stop accuracy (per station); 40% timetable adherence; 20% no penalties

R13.04.01.01-T15 Load passengers and fuel so the C172 is within the CG envelope

Original instruction

FlightGear c172p: using the Weight and Balance dialog, set pilot 180 lb, front passenger 160 lb, rear passengers and baggage per input/load.json, then adjust fuel so total weight is ≤ 2,450 lb and CG is within the normal category envelope. Report weight and CG in answer.json.

Platforms: mac · macOS win · Windows lin · Linux
Applications: FlightGear
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: click · Point & click type · Text & forms
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Property tree: payload and fuel values; computed weight ≤ 2,450 lb and CG inside the POH envelope; answer.json within ±5 lb and ±0.1 in of the computed values. The telnet/HTTP property server is not reachable from the agent's VM, and the in-sim property browser and Nasal console are disabled, so loads must be set in the dialog.

Proposed partial credit

40% payload entered; 30% within envelope; 30% answer correct

R13.04.01.01-T16 Tune radios and set the transponder in Microsoft Flight Simulator

Original instruction

MSFS, Cessna 172 (G1000) cold on the ramp at KPAO with avionics on. Set COM1 active to Palo Alto ground 125.000, standby to tower 118.600, NAV1 to SJC VOR 114.100, transponder to 4521 in ALT mode, and altimeter to 29.92 inHg using the cockpit knobs.

Platforms: win · Windows
Applications: MSFS
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: AP · API-sufficient
Interaction primitives: click · Point & click drag · Drag & drop vis · Visual judgment
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

SimConnect variables: COM1 active/standby, NAV1 active, transponder code and mode, Kollsman setting, all exact. External tools (radio panels, apps) are not connected, and SimConnect is not exposed to the agent (only the harness client is accepted), so radios cannot be set by script.

Proposed partial credit

40% COM1 active and standby; 20% NAV1; 25% transponder; 15% altimeter

R13.04.01.01-T17 Berth a vessel alongside a pier below one knot in Bridge Command

Original instruction

Bridge Command (ship simulator, desktop) scenario: a 60 m vessel with twin engines and bow thruster approaches a pier. Using the engine, rudder and thruster controls, bring the ship alongside the marked berth with speed over ground < 1 kt and lateral speed < 0.3 kt at contact.

Platforms: mac · macOS win · Windows lin · Linux
Applications: Bridge Command
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control click · Point & click vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Scenario log (NMEA output captured by the harness): final position within the berth polygon, SOG < 1 kt and lateral speed < 0.3 kt at first contact; no collision with other objects. The simulator's network input port is not exposed to the agent and it has no shell, so the vessel cannot be steered by script.

Proposed partial credit

40% inside berth; 30% speed limits; 30% no collisions

R13.04.01.01-T18 Keep lane on a highway at 90 km/h in CARLA

Original instruction

CARLA Town04 highway, manual control. Accelerate to 90 ±5 km/h and keep the vehicle centred in lane 2 for 2 minutes through curves, with no lane departures.

Platforms: lin · Linux win · Windows
Applications: CARLA
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Recorder: speed within band for ≥ 90% of the window; lateral deviation RMS ≤ 0.3 m; lane-invasion sensor events = 0. The CARLA Python client port is firewalled from the agent VM, so the ego vehicle cannot be driven by script; control comes only from the manual-control window.

Proposed partial credit

30% speed in band; 50% scaled by RMS deviation (1.0 → 0.3 m); 20% no lane invasions

R13.04.01.01-T19 Reach a circular 80 km orbit in Kerbal Space Program

Original instruction

Kerbal Space Program (1.12) loads a sandbox save with a pre-built two-stage rocket on the launchpad. Launch, perform a gravity turn and circularise into an orbit with periapsis and apoapsis both 75-85 km. Use the navball and map view; MechJeb and other autopilot mods are not installed.

Platforms: mac · macOS win · Windows lin · Linux
Applications: Kerbal Space Program
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: rt · Real-time control key · Shortcuts vis · Visual judgment
Work product: WP11 · Game/sim state
Conditions: None declared
Proposed difficulty: 5 / 5

Proposed verifier

Persistent save (persistent.sfs) parsed: vessel orbit around Kerbin with PeA and ApA in 75-85 km; vessel intact. Debug menu (Alt+F12) disabled.

Proposed partial credit

20% reach space (70 km); 30% stable orbit (PeA > 70 km); 50% scaled by circularity within 75-85 km

R13.04.01.01-T20 Program a 737 FMC for departure in X-Plane

Original instruction

X-Plane 12 default Boeing 737-800 cold at KSEA gate. Using the CDU, enter route from input/ofp.pdf (SID, airways, STAR into KSFO), performance data (ZFW, fuel, cost index, cruise FL350), compute and set V-speeds, and arm LNAV/VNAV for takeoff. Do not take off.

Platforms: mac · macOS win · Windows lin · Linux
Applications: X-Plane
Interfaces: desk · Desktop GUI
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms vis · Visual judgment
Work product: WP08 · App/system state
Conditions: long · Long-horizon tutorial · Follow docs/tutorial
Proposed difficulty: 4 / 5

Proposed verifier

FMS datarefs via harness plugin: route legs equal the OFP; ZFW, fuel and CI within tolerance; cruise altitude 35,000; V1/VR/V2 set (any values the FMC computed); LNAV and VNAV armed. X-Plane's UDP port is firewalled from the agent VM and no FlyWithLua/XPPython3 plugins are installed, so the FMS cannot be loaded by script.

Proposed partial credit

40% route; 20% performance data; 20% V-speeds; 20% LNAV/VNAV armed

Mapped research references

No benchmark reference is assigned to this class in the current mapping.