Performance profiling & optimization
Profile CPU, memory or rendering, find the hotspot and speed it up to a target.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Instruments · perf · Chrome DevTools · VTune
- Class GUI-need label
HY· Hybrid
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R02.03.01.02-T01 Profile and speed up a Python report generator 3x
Original instruction
`/work/salesreport` takes ~60 s to build `report.xlsx` from `fixtures/orders.parquet`. Profile it (cProfile + snakeviz or py-spy), save the profile to `prof/before.prof`, and optimize the hotspots so the run is at least 3x faster with an identical workbook. Save `prof/after.prof`.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- cProfile · snakeviz · py-spy
- Interfaces
term· Terminalbrow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader box timing (median of 5): speedup >= 3x vs baseline; output workbook cell values and formats identical (openpyxl diff); both .prof files load and after.prof total time lower; hidden tests green.
Proposed partial credit
Linear 1x->3x speedup (60%); 30% identical output; 10% profiles saved
R02.03.01.02-T02 Remove scrolling jank in a macOS app using Instruments Time Profiler
Original instruction
The macOS app `PhotoGrid` (Xcode project) stutters when scrolling a 5,000-item grid. Profile with Instruments (Time Profiler + Animation Hitches), save the trace to `traces/before.trace`, fix the main-thread hotspots, and record `traces/after.trace` showing hitches resolved.
- Platforms
mac· macOS- Applications
- Instruments · Xcode
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden XCUITest scroll benchmark (XCTOSSignpostMetric scrollDecelerationMetric): hitch time ratio < 5 ms/s on grader Mac (baseline ~40); both .trace bundles exist and after shows lower main-thread sample count; UI snapshot tests unchanged.
Proposed partial credit
50% hitch target (linear); 20% visual output unchanged; 30% traces recorded before/after
R02.03.01.02-T03 Find a C++ service hotspot with perf and flame graphs, then fix it
Original instruction
`/work/geoindex` (C++) answers range queries at 12k qps; the target is 18k qps. Use `perf record -g` and generate a flame graph (`flame.svg`) to find the hotspot, then optimize it (algorithm or data layout), keeping results identical.
- Platforms
lin· Linux- Applications
- perf · FlameGraph · CMake
- Interfaces
term· Terminalbrow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader benchmark on pinned cores: >= 18k qps (linear credit from 12k); query results identical on 100k queries; flame.svg present and generated from perf data of this binary (embedded symbol names).
Proposed partial credit
60% throughput (linear); 30% identical results; 10% flame graph
R02.03.01.02-T04 Cut long tasks on a page to meet an INP budget using the Performance panel
Original instruction
The sandbox product-filter page in `/work/shop-web` has INP around 600 ms when toggling filters on a mid-tier CPU (4x throttling). Use the Chrome DevTools Performance panel to find long tasks, save the profile JSON, and optimize so INP < 200 ms under the same throttling, with identical filtering results.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Chrome DevTools · Vite
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickvis· Visual judgmentcode· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden Playwright + web-vitals run with CPU 4x throttle: p75 INP over 30 interactions < 200 ms; DOM results identical to reference for 20 filter combos; saved profile JSON shows the original long task.
Proposed partial credit
60% INP target (linear 600->200); 30% identical results; 10% profile saved
R02.03.01.02-T05 Vectorize a memory-bound loop guided by Intel VTune
Original instruction
`/work/stencil3d` (C++/OpenMP) runs a 7-point stencil at 30 GFLOP/s on the grader's Xeon. Use VTune (Hotspots, Memory Access and Roofline) to diagnose, save the result directory, and restructure (tiling, vectorization, alignment) to reach >= 60 GFLOP/s with results within 1e-12 of baseline.
- Platforms
lin· Linuxwin· Windows- Applications
- Intel VTune · Intel Advisor · icx/gcc
- Interfaces
desk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader run: GFLOP/s >= 60 (linear credit from 30); max abs diff <= 1e-12 on 3 grids; VTune result dir present with memory-access analysis of the baseline.
Proposed partial credit
60% throughput (linear); 30% numerics; 10% analysis evidence
R02.03.01.02-T06 Halve app startup time on Windows using WPR/WPA
Original instruction
`C:\work\Planner` (.NET WPF) takes ~4.2 s to show its main window. Record a startup trace with Windows Performance Recorder, analyze in Windows Performance Analyzer (CPU usage sampled, disk I/O, file I/O), save the .etl, and reduce time-to-main-window below 2.0 s.
- Platforms
win· Windows- Applications
- WPR · WPA · Visual Studio
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader measures time from process start to main window ETW event (median of 10 cold starts): < 2.0 s (linear from 4.2); functional UI smoke test passes; startup.etl present.
Proposed partial credit
60% startup target (linear); 30% functionality; 10% trace saved
R02.03.01.02-T07 Unblock a Node event loop identified with clinic.js
Original instruction
`/work/api-node` p99 latency spikes to 900 ms under load. Run clinic doctor and flame (or 0x), save the HTML reports to `perf/`, find the synchronous work blocking the event loop, and fix it so p99 < 120 ms at 500 rps with identical responses.
- Platforms
lin· Linuxmac· macOS- Applications
- clinic.js · autocannon · Node
- Interfaces
term· Terminalbrow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader autocannon run (500 rps, 60 s): p99 < 120 ms (linear credit from 900); response bodies identical on 1,000 sampled requests; event-loop delay max < 50 ms; perf/ reports present.
Proposed partial credit
60% latency target; 25% identical responses; 15% reports
R02.03.01.02-T08 Halve allocations in a .NET service with the Visual Studio profiler
Original instruction
`C:\work\Quotes` (.NET 8 API) allocates ~1.4 GB/min under the provided load script, causing gen2 GCs. Use the Visual Studio .NET Object Allocation tool, save the `.diagsession`, and reduce allocations by >= 50% without changing outputs.
- Platforms
win· Windows- Applications
- Visual Studio Profiler · dotnet-counters
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
dotnet-counters during grader load: alloc rate <= 50% of baseline and gen2 count reduced; response bodies identical; BenchmarkDotNet hidden benchmarks no slower; diagsession present.
Proposed partial credit
60% allocation reduction (linear up to halving); 25% identical outputs; 15% artifacts
R02.03.01.02-T09 Speed up a slow reporting SQL query 10x using EXPLAIN ANALYZE
Original instruction
In the Postgres DB from `/work/analytics-db`, the query in `slow.sql` takes ~14 s. Use EXPLAIN (ANALYZE, BUFFERS) to find the bottleneck, then rewrite the query and/or add indexes (migration file in `migrations/`) so it runs in < 1.4 s with the exact same result set.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Postgres · psql
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs new query 5 times on a fresh DB with migration applied: median < 1.4 s; result set identical (ordered hash); other benchmark queries not slower by > 10%; index count added <= 3.
Proposed partial credit
60% speed (linear); 30% identical result; 10% no regressions
R02.03.01.02-T10 Fix a detached-DOM memory leak using heap snapshots
Original instruction
The sandbox dashboard SPA in `/work/ops-dash` grows ~30 MB per route change. Use Chrome DevTools Memory panel heap snapshots (compare before/after 10 navigations), save both `.heapsnapshot` files, find the retainers of detached DOM trees, and fix the leak.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Chrome DevTools
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden Playwright test: JS heap after 100 route changes and forced GC within 5 MB of after 10; detached node count < 50; app behavior tests pass; snapshot files present and contain the leaking class name.
Proposed partial credit
60% leak fixed; 25% functionality; 15% snapshots
R02.03.01.02-T11 Find a memory growth in a Mac app with Instruments Allocations and Leaks
Original instruction
The macOS app `MailPreview` grows 200 MB per hour while idle with an account syncing. Use Instruments Allocations (generations/mark heap) and Leaks, save the trace, identify the retain cycle or unbounded cache, and fix it so memory stays flat over a 30-minute accelerated sync test.
- Platforms
mac· macOS- Applications
- Instruments · Xcode
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden test runs accelerated sync 30 min: footprint growth < 10 MB; Leaks reports none; trace saved with a generation analysis; UI tests unchanged.
Proposed partial credit
60% memory flat; 20% no leaks; 20% trace evidence
R02.03.01.02-T12 Profile a live Python web service with py-spy and fix the hotspot
Original instruction
The Flask service in `/work/geocoder` (running under gunicorn in Docker) uses 100% CPU at 50 rps. Without restarting it, attach `py-spy record` to a worker for 60 s and save `profile.svg`, then fix the hotspot so CPU at 50 rps is < 30% with identical responses.
- Platforms
lin· Linux- Applications
- py-spy · Docker · gunicorn
- Interfaces
term· Terminalbrow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader load test at 50 rps: container CPU < 30% (linear from 100%); responses identical for 2,000 requests; profile.svg present, generated by py-spy from a live PID (metadata).
Proposed partial credit
60% CPU target; 30% identical responses; 10% profile
R02.03.01.02-T13 Overlap CUDA transfers and kernels guided by Nsight Systems
Original instruction
`/work/batchinfer` (CUDA C++) processes 1,000 image batches in 9.5 s on the grader GPU with idle gaps. Capture a timeline with Nsight Systems (`.nsys-rep`), identify serialization between H2D copies, kernels and D2H, and restructure with streams/pinned memory to finish in < 5 s with identical outputs.
- Platforms
lin· Linux- Applications
- Nsight Systems · nvcc
- Interfaces
desk· Desktop GUIterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Grader run: wall time < 5 s (linear from 9.5); output tensor checksums identical; nsys report shows >= 2 concurrent streams with overlapping memcpy and kernel ranges.
Proposed partial credit
60% time target; 25% identical outputs; 15% overlap evidenced in timeline
R02.03.01.02-T14 Remove frame jank in an Android list with the Android Studio profiler
Original instruction
The `NewsFeed` Android app janks while scrolling (frames > 16 ms). Use Android Studio's CPU profiler and System Trace, save the trace, find the main-thread work (image decoding, layout passes), and fix it so > 95% of frames render under 16 ms on the emulator benchmark.
- Platforms
mac· macOSwin· Windowslin· Linuxand· Android- Applications
- Android Studio Profiler · Macrobenchmark
- Interfaces
desk· Desktop GUImob· Mobile touch- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden Macrobenchmark FrameTimingMetric: p95 frame duration < 16 ms (baseline 38); list content screenshots identical; trace file present.
Proposed partial credit
60% frame target; 25% visual parity; 15% trace
R02.03.01.02-T15 Raise a marketing page's Lighthouse performance score to 90+
Original instruction
The sandbox landing page `/work/landing` scores 41 on Lighthouse mobile. Use Lighthouse and the DevTools Performance/Network panels to find the causes (render-blocking CSS, unoptimized hero image, layout shift, third-party script), fix them, and reach >= 90 without changing the visible design.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Lighthouse · Chrome DevTools
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickvis· Visual judgmentcode· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader Lighthouse CI (3 runs, median, mobile): performance >= 90 (linear from 41); CLS < 0.1; full-page screenshot SSIM >= 0.97 vs original design at 3 viewports.
Proposed partial credit
60% score (linear); 15% CLS; 25% design unchanged
R02.03.01.02-T16 Reduce allocations in a Go service using pprof
Original instruction
`/work/logagg` (Go) spends 40% of CPU in GC. Capture CPU and alloc profiles via net/http/pprof under the load script, inspect with `go tool pprof -http`, save `cpu.pb.gz` and `allocs.pb.gz`, and cut allocations per request by >= 60%.
- Platforms
lin· Linuxmac· macOS- Applications
- pprof · go
- Interfaces
term· Terminalbrow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden benchmark: allocs/op <= 40% of baseline; ns/op not worse; output identical on corpus; profile files present and parseable.
Proposed partial credit
60% alloc target (linear); 15% speed not worse; 15% identical output; 10% profiles
R02.03.01.02-T17 Double the throughput of a Rust parser measured with criterion
Original instruction
Crate `/work/fastcsv` parses at 450 MB/s on the criterion benchmark. Profile with `cargo flamegraph` or samply, then optimize (SIMD scanning, fewer bounds checks, buffer reuse) to >= 900 MB/s while remaining safe Rust except for documented, tested `unsafe` blocks.
- Platforms
lin· Linuxmac· macOS- Applications
- cargo · criterion · samply
- Interfaces
term· Terminalbrow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commandsvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader criterion run on pinned CPU: >= 900 MB/s (linear from 450); differential fuzz vs baseline 10 min no mismatch; every unsafe block has a SAFETY comment and Miri passes on unit tests.
Proposed partial credit
60% throughput; 25% correctness fuzz; 15% unsafe hygiene
R02.03.01.02-T18 Fix the top query by pg_stat_statements in a Postgres app
Original instruction
The sandbox app DB has `pg_stat_statements` enabled after a day of synthetic traffic. Identify the query with the highest total execution time, write its normalized text to `TOP.sql`, and add the single index that best fixes it (migration in `migrations/`), verifying with EXPLAIN.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Postgres · psql · pgAdmin
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
TOP.sql matches the seeded top query (normalized); after migration, that query's mean time drops >= 10x in the replay; exactly one index added; replay of all other queries not slower than 10%.
Proposed partial credit
40% correct query; 50% speedup; 10% single index
R02.03.01.02-T19 Bring a Unity scene's frame time under 16.6 ms using the Profiler
Original instruction
In the Unity project `/work/CityDemo`, the `Downtown` scene runs at ~38 ms/frame on the grader GPU. Use the Unity Profiler and Frame Debugger to find the costs (draw calls, GC allocs in Update, expensive shadows), save the profiler capture, and reach a mean frame time < 16.6 ms on the scripted camera path without visible changes beyond tolerance.
- Platforms
win· Windowsmac· macOS- Applications
- Unity Editor · Unity Profiler
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
HY· Hybrid- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
licensed· Licensed pro app- Proposed difficulty
- 5 / 5
Proposed verifier
Grader player build runs the camera path: mean frame time < 16.6 ms (linear from 38), GC alloc per frame < 1 KB; 10 checkpoint frames SSIM >= 0.95 vs original; .data profiler capture present. Unity -batchmode launches and Profiler.logFile/Profiler API capture scripts are detected (process audit, script diff) and void the capture milestone.
Proposed partial credit
50% frame time; 15% GC allocs; 25% visual parity; 10% capture
R02.03.01.02-T20 Cut an Electron app's idle CPU from 12% to under 2%
Original instruction
The Electron app `/work/notes-desktop` uses ~12% CPU while idle. Using Chrome DevTools for renderer and main processes plus the OS profiler (Instruments on macOS or WPA on Windows), find all idle work sources (timers, animations, IPC polling, a spinner hidden offscreen), save the traces, and reduce idle CPU below 2% on both OSes.
- Platforms
mac· macOSwin· Windows- Applications
- Chrome DevTools · Instruments · WPA
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP06· Code/engineering- Conditions
multiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Grader measures idle CPU over 5 min after 1 min settle on mac and Windows VMs: < 2% both (linear from 12); functional E2E tests pass; traces from both OS profilers present.
Proposed partial credit
35% macOS target; 35% Windows target; 20% functionality; 10% traces
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.