IDE & editor setup and operations
Set up and operate IDEs and editors: create, open and arrange files and workspaces, use editor commands (multi-line indent, multi-cursor, find and replace), and configure extensions, settings, run/debug configurations and keymaps.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- VS Code · JetBrains IDEs · Xcode · Visual Studio
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R02.03.01.03-T01 VS Code compound debug config for API and browser
Original instruction
Workspace ~/work/shop (Node API in api/, React app in web/). In VS Code create debug configurations: 'API' launching api/src/server.ts via tsx with env PORT=4000 and a preLaunchTask 'build:api', 'Web' launching Chrome at http://localhost:5173 with webRoot web/src, and a compound 'Full stack' starting both and stopping all on exit.
- Platforms
mac· macOSwin· Windows- Applications
- VS Code
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader parses .vscode/launch.json and tasks.json (schema + fields), then starts 'Full stack' through the VS Code extension-test harness and checks a breakpoint in api/src/routes.ts:42 and web/src/App.tsx:17 binds. File provenance: writes to .vscode/*.json by any process other than Code are detected (fanotify/eslogger) -> 0.
Proposed partial credit
30% API config incl. env and preLaunchTask; 25% Web config; 20% compound with stopAll; 25% both breakpoints bind in a real session
R02.03.01.03-T02 Rebind three VS Code shortcuts without conflicts
Original instruction
In VS Code's Keyboard Shortcuts editor: bind Ctrl+Alt+T (Cmd+Alt+T on mac) to 'Terminal: Create New Terminal' only when the terminal is not focused, remove the default binding of 'Trim Trailing Whitespace' (Ctrl/Cmd+K Ctrl/Cmd+X), and add chord Ctrl/Cmd+K Ctrl/Cmd+G for 'Go to Line'.
- Platforms
mac· macOSwin· Windows- Applications
- VS Code
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickkey· Shortcutstype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
User keybindings.json contains exactly 3 entries: workbench.action.terminal.new with when '!terminalFocus', a '-editor.action.trimTrailingWhitespace' removal, and the chord for workbench.action.gotoLine; no other user bindings. Writes to keybindings.json by processes other than Code are detected (file provenance) -> 0.
Proposed partial credit
40% terminal binding with correct when clause; 30% default removed; 30% chord added
R02.03.01.03-T03 PyCharm interpreter, source roots and pytest run config
Original instruction
Open ~/work/pricing in PyCharm. Set the project interpreter to the existing .venv, mark src as Sources Root and tests as Test Sources Root, set the default test runner to pytest, and add a run configuration 'unit' running pytest on tests/unit with env PRICING_ENV=test and working directory = project root. Share the run configuration.
- Platforms
mac· macOSwin· Windows- Applications
- PyCharm
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
.idea/misc.xml sdk name points at .venv; .iml sourceFolder entries (src isTestSource=false, tests true); TestRunnerService=py.test; .run/unit.run.xml has target tests/unit, env and workdir; grader runs it headless and gets 37 tests passing. .idea writes by non-IDE processes are detected (file provenance) -> 0.
Proposed partial credit
25% interpreter; 25% source and test roots; 15% pytest default; 35% shared run config works
R02.03.01.03-T04 Xcode Staging configuration and scheme
Original instruction
In Xcode open WeatherApp.xcodeproj. Duplicate the Release build configuration as 'Staging', create a shared scheme 'WeatherApp-Staging' whose Run and Archive actions use Staging, add SWIFT_ACTIVE_COMPILATION_CONDITIONS 'STAGING' for Staging only, and set the scheme's Run environment variable API_BASE=https://staging.api.test.
- Platforms
mac· macOS- Applications
- Xcode
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
project.pbxproj has XCBuildConfiguration 'Staging' for project and app target with STAGING flag only there; xcshareddata/xcschemes/WeatherApp-Staging.xcscheme LaunchAction and ArchiveAction buildConfiguration=Staging with the env var; xcodebuild -scheme ... -showBuildSettings confirms. pbxproj writes by non-Xcode processes are detected (eslogger) -> 0.
Proposed partial credit
35% Staging configuration on both levels; 20% compilation condition scoped; 30% shared scheme actions; 15% env var
R02.03.01.03-T05 Visual Studio multiple startup projects and launch profile
Original instruction
In Visual Studio 2022 open Orders.sln. Set multiple startup projects: Orders.Api (Start) and Orders.Worker (Start), Orders.Tests (None). In Orders.Api add launch profile 'Dev-5080' (Project, applicationUrl http://localhost:5080, ASPNETCORE_ENVIRONMENT=Development) and make it active. Enable 'Run code analysis on build' for Orders.Api.
- Platforms
win· Windows- Applications
- Visual Studio
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Startup projects live in the binary .vs/Orders/v17/.suo (GUI-only property): grader parses the OLE stream for the 2 Start entries; launchSettings.json profile fields exact; Orders.Api.csproj RunAnalyzersDuringBuild=true. Grader F5-launches via DTE and checks both processes start.
Proposed partial credit
40% multiple startup set; 30% launch profile exact and active; 15% code analysis on build; 15% both start
R02.03.01.03-T06 Offline extension install and per-workspace disable
Original instruction
Without internet, install the VS Code extensions from fixtures/vsix/ (ESLint 3.0.10 and Prettier 11.0.0), disable GitLens for this workspace only (keep it enabled globally), and set Prettier as the default formatter for TypeScript in workspace settings with format-on-save.
- Platforms
mac· macOSwin· Windows- Applications
- VS Code
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickdrag· Drag & drop- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
code --list-extensions --show-versions (grader) has both exact versions; workspace state.vscdb (SQLite) extensionsIdentifiers/disabled lists GitLens only for this workspace; .vscode/settings.json [typescript] defaultFormatter and formatOnSave. Agent terminal withheld; `code --install-extension` exec is detected -> 0.
Proposed partial credit
35% both VSIX versions installed; 35% GitLens disabled only in workspace; 30% formatter settings
R02.03.01.03-T07 IntelliJ project code style with import layout
Original instruction
In IntelliJ IDEA for ~/work/ledger (Java), create a project code style scheme: 4-space indent, continuation indent 8, no tabs, right margin 120, class count to use import with '*' 99, imports layout: static imports last, java.* and javax.* after others, blank line between groups. Then reformat src/main/java/ledger/Entry.java.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- IntelliJ IDEA
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & formsdrag· Drag & drop- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
.idea/codeStyles/Project.xml and codeStyleConfig.xml (USE_PER_PROJECT_SETTINGS) contain the values; grader runs IntelliJ's headless formatter with that scheme on 5 hidden files and compares to expected output; Entry.java equals expected. .idea writes by non-IDE processes detected -> 0.
Proposed partial credit
35% indent and margin values; 30% import layout; 35% hidden-file formatting matches
R02.03.01.03-T08 VS Code profile for docs work bound to a workspace
Original instruction
Create a VS Code profile 'Docs' containing only the extensions Markdown All in One and Code Spell Checker (from fixtures/vsix), with settings editor.wordWrap 'on' and cSpell.language 'en,de'. Associate the workspace ~/work/handbook with the Docs profile so it opens with it, while other folders keep the Default profile.
- Platforms
mac· macOSwin· Windows- Applications
- VS Code
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
User data: profiles/<id>/extensions.json has exactly the 2 extensions; profiles/<id>/settings.json values; storage.json profileAssociations maps the handbook folder URI to Docs. Grader opens another folder and checks Default profile. Profile writes by non-Code processes detected -> 0.
Proposed partial credit
30% profile with exact extensions; 25% profile settings; 45% workspace association works
R02.03.01.03-T09 PyCharm Docker Compose interpreter for a service
Original instruction
In PyCharm Professional, configure the project interpreter for ~/work/shopapi as a Docker Compose interpreter using docker-compose.yml service 'web' (Python at /usr/local/bin/python), with path mapping project root -> /app. Then create a Django server run configuration using it on 0.0.0.0:8000.
- Platforms
mac· macOSwin· Windows- Applications
- PyCharm · Docker Desktop
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro appmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
.idea/misc.xml + jdk.table.xml (options dir) define a docker-compose remote SDK for service web; run config uses it with host 0.0.0.0 port 8000; grader starts it via the IDE and gets HTTP 200 from /health. Config writes by non-IDE processes detected (file provenance) -> 0.
Proposed partial credit
45% compose interpreter for service web; 20% path mapping; 35% run config serves /health
R02.03.01.03-T10 Problem matcher for a custom compiler
Original instruction
Workspace ~/work/fwbuild builds with tools/fwcc, whose error format is shown in fixtures/fwcc_sample.log. Create a VS Code build task 'fw: build' (default build task) with a problem matcher that captures file, line, column, severity and message, using the workspace folder as base path.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- VS Code
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & formscode· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
tasks.json: task is group build isDefault; grader applies the matcher regex to a hidden 30-line log and must extract exactly 7 problems with correct fields; grader runs the task via the VS Code test harness and counts Problems = 7. tasks.json writes by non-Code processes detected -> 0.
Proposed partial credit
20% default build task; 50% matcher extracts all fields on hidden log; 30% Problems panel count correct
R02.03.01.03-T11 Xcode signing, bundle ID and capabilities
Original instruction
In Xcode, for target 'NotesApp' set automatic signing with team 'QA Test Team', bundle identifier com.example.notes.qa, add capabilities Push Notifications and App Groups with group.com.example.notes.shared, and enable Background Modes > Remote notifications.
- Platforms
mac· macOS- Applications
- Xcode
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
pbxproj: CODE_SIGN_STYLE Automatic, DEVELOPMENT_TEAM matches, PRODUCT_BUNDLE_IDENTIFIER exact; NotesApp.entitlements has aps-environment and the app group; Info.plist UIBackgroundModes contains remote-notification. pbxproj/entitlements writes by non-Xcode processes detected (eslogger) -> 0.
Proposed partial credit
30% signing and team; 15% bundle ID; 35% push and app group entitlements; 20% background mode
R02.03.01.03-T12 Visual Studio keyboard scheme and exported settings
Original instruction
In Visual Studio 2022 switch the keyboard mapping scheme to 'Visual Studio Code', then in Text Editor scope bind Edit.ToggleLineComment to Ctrl+/ (removing conflicting bindings in that scope) and Edit.Duplicate to Shift+Alt+Down. Export only keyboard settings to C:\out\keys.vssettings.
- Platforms
win· Windows- Applications
- Visual Studio
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickkey· Shortcutstype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
keys.vssettings XML: ShortcutsScheme = Visual Studio Code; UserShortcuts contain the two bindings in Text Editor scope and Remove entries for conflicts; export contains only the Keyboard category. Grader imports into a clean VS and sends Ctrl+/ via UIA to toggle a comment. Hand-written .vssettings detected by missing VS export header -> 0.
Proposed partial credit
25% scheme set; 45% two bindings with conflicts removed; 30% export scoped and functional
R02.03.01.03-T13 Python live templates with variables in PyCharm
Original instruction
Create a live template group 'team' in PyCharm with three Python templates from fixtures/templates.md: 'tdef' (pytest function with $NAME$ and $END$), 'logr' (module logger using fileNameWithoutExtension()), and 'dcls' (dataclass with frozen=True and a field placeholder). Export them via File > Manage IDE Settings > Export to ~/out/team-templates.zip.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- PyCharm
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Exported ZIP templates/team.xml: 3 templates with exact text, variables, expressions and Python context; grader imports into a clean IDE, expands each template in a test file via the IDE driver and compares output. ZIP must carry the IDE export manifest (hand-made ZIP detected) -> 0.
Proposed partial credit
30% three templates exist with context; 40% variables and expressions correct; 30% export imports and expands
R02.03.01.03-T14 Dev Container configuration and reopen in container
Original instruction
For ~/work/etl in VS Code, use 'Dev Containers: Add Dev Container Configuration Files' to create a Python 3.12 container, add features node (lts) and github-cli, forward port 8888, set postCreateCommand 'pip install -r requirements.txt', and reopen the folder in the container.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- VS Code · Dev Containers · Docker Desktop
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
.devcontainer/devcontainer.json fields (image tag 3.12, two features, forwardPorts, postCreateCommand); docker ps shows a container labelled devcontainer.local_folder=<path> running; inside it python 3.12, node, gh present and requirements installed.
Proposed partial credit
40% devcontainer.json exact; 35% container built and running; 25% tools and requirements installed
R02.03.01.03-T15 Conditional, logging and exception breakpoints in PyCharm
Original instruction
In ~/work/orders (PyCharm): add a Python exception breakpoint for ValueError limited to project code (not library files); a line breakpoint at app/service.py:88 with condition 'order.total > 1000'; and a non-suspending breakpoint at app/service.py:120 that logs 'paid {order.id}'. Run the 'orders' debug config and save the log output to ~/out/paid.log.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- PyCharm
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
workspace.xml XDebuggerManager: exception breakpoint ValueError with ignore-library-files, condition and log expression exact, suspend=NONE; paid.log contains the 6 expected ids in order. Breakpoint state exists only in the IDE workspace; writes by non-IDE processes detected -> 0.
Proposed partial credit
25% exception breakpoint scoped; 25% conditional breakpoint; 20% logging breakpoint non-suspending; 30% log output correct
R02.03.01.03-T16 Multi-root workspace with per-folder interpreters
Original instruction
Create a VS Code multi-root workspace ~/work/mono.code-workspace containing services/auth, services/billing and libs/common. Set python.defaultInterpreterPath per folder to each folder's .venv, add search.exclude and files.watcherExclude for **/node_modules and **/.venv at workspace level, and name the folders 'Auth', 'Billing', 'Common'.
- Platforms
mac· macOSwin· Windows- Applications
- VS Code
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & formsdrag· Drag & drop- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
.code-workspace JSON: 3 folders with names; workspace settings excludes; each folder's .vscode/settings.json interpreter path; grader opens it in VS Code and the Python extension reports the right interpreter per folder. Writes by non-Code processes detected (file provenance) -> 0.
Proposed partial credit
30% workspace folders and names; 30% excludes; 40% per-folder interpreters resolved
R02.03.01.03-T17 Shared Xcode breakpoints for layout and exceptions
Original instruction
In Xcode project LayoutKit.xcodeproj add a symbolic breakpoint on UIViewAlertForUnsatisfiableConstraints with a debugger command action 'po $arg1' and auto-continue off, plus an Objective-C exception breakpoint on Throw. Share both breakpoints with the project.
- Platforms
mac· macOS- Applications
- Xcode
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
xcshareddata/xcdebugger/Breakpoints_v2.xcbkptlist contains a SymbolicBreakpoint with symbol exact and DebuggerCommand action, and an ExceptionBreakpoint scope objc, catch on throw. Writes by non-Xcode processes are detected (eslogger) -> 0.
Proposed partial credit
50% symbolic breakpoint with action; 30% exception breakpoint; 20% shared location
R02.03.01.03-T18 CMake preset with AddressSanitizer in Visual Studio
Original instruction
Open the CMake project C:\work\imgtool in Visual Studio 2022. Add a configure preset 'x64-asan' (inherits x64-debug) enabling AddressSanitizer through Visual Studio's CMake settings, with build directory out/build/asan, select it, build, and run the 'imgtool_tests' target so the seeded heap overflow is reported.
- Platforms
win· Windows- Applications
- Visual Studio · CMake
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
CMakePresets.json (or CMakeUserPresets) has x64-asan inheriting x64-debug with the ASan flag and binaryDir; out/build/asan/CMakeCache.txt shows /fsanitize=address; test output in the VS Output log contains 'heap-buffer-overflow' in decode.cpp. Preset writes by non-devenv processes detected (Sysmon) -> 0.
Proposed partial credit
35% preset correct; 30% configured and built with ASan; 35% overflow reported
R02.03.01.03-T19 Offline extensions via CLI (control pair of T06)
Original instruction
Control pair of T06: same VSIX files and workspace, but use only the terminal: install both VSIX files with the code CLI, write the workspace settings file, and disable GitLens for the workspace if a CLI route exists; write which parts were impossible from the CLI to ~/out/cli_gaps.txt.
- Platforms
mac· macOSwin· Windows- Applications
- VS Code CLI · bash
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Same extension and settings checks as T06; cli_gaps.txt must state that per-workspace disable has no supported CLI flag (grader checks the claim against state.vscdb: GitLens still enabled is expected).
Proposed partial credit
45% both VSIX installed; 30% settings exact; 25% CLI gap reported correctly
R02.03.01.03-T20 Full IntelliJ onboarding setup for a Gradle monorepo
Original instruction
Set up ~/work/fleet in IntelliJ IDEA for the team: Gradle JVM = bundled JDK 21, project SDK 21, import fixtures/inspections.xml as the project inspection profile and raise 'Unused declaration' to Error, add a file header template from fixtures/header.txt, create 3 shared run configs from fixtures/runs.md, and commit .idea files listed in fixtures/vcs_allow.txt.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- IntelliJ IDEA · Gradle · git
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP06· Code/engineering- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
gradle.xml gradleJvm, misc.xml SDK; inspectionProfiles/Project_Default.xml with severity change; fileTemplates/includes/File Header.java; .run/*.run.xml 3 configs run headless successfully; git commit contains exactly the allowed .idea files. .idea writes by non-IDE processes detected -> 0.
Proposed partial credit
25% SDK, Gradle JVM and header template; 25% inspection profile; 30% three run configs work; 20% commit scope exact
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.