Exploratory & statistical analysis
Wrangle, analyze and model data in notebooks to a checked answer.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Jupyter · pandas · R
- Class GUI-need label
CL· CLI-sufficient
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R03.02.01.01-T01 Analyze an A/B test with CUPED variance reduction
Original instruction
data/ab_checkout.parquet holds a 2-week checkout experiment (user, variant, revenue, pre-period revenue). In a Jupyter notebook, compute the relative lift in revenue per user with a 95% CI, unadjusted and CUPED-adjusted, check for sample-ratio mismatch, and write output/result.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · pandas · SciPy
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader compares result.json to gold: lift ±0.001, CI bounds ±0.002, CUPED theta ±0.01, SRM p-value ±0.005 and srm_flag boolean exact. Notebook must execute top to bottom with nbconvert and reproduce the JSON.
Proposed partial credit
25% unadjusted lift/CI; 35% CUPED estimate/CI; 20% SRM check; 20% notebook reproduces
R03.02.01.01-T02 Estimate price elasticity with fixed effects
Original instruction
Using data/store_weekly.parquet (SKU × store × week units and price), estimate own-price elasticity with a log-log regression including SKU, store and week fixed effects and clustered standard errors by store. Report the elasticity, SE and 95% CI in output/elasticity.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · statsmodels · pandas
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Gold from a reference fixest/linearmodels fit: elasticity ±0.005, clustered SE ±5%, CI consistent. Grader reruns the notebook on a hidden resample with known true elasticity −1.4: estimate within ±0.05.
Proposed partial credit
40% point estimate; 25% clustered SE; 15% CI; 20% hidden resample recovery
R03.02.01.01-T03 Compare time-to-churn by plan with survival analysis in R
Original instruction
data/subscriptions.csv has start, end (censored if active) and plan. In R, fit Kaplan-Meier curves by plan, report median survival with 95% CI, run a log-rank test, fit a Cox model with plan and region, and check proportional hazards. Save output/survival.json and output/km.png.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- R · RStudio · survival
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Gold from survival::survfit/coxph: medians ±1 day, CIs ±2 days, log-rank chi-square ±0.1, hazard ratios ±0.01, PH test p-values ±0.01. km.png must exist with 3 curves (VLM check, 10%).
Proposed partial credit
30% KM medians/CIs; 15% log-rank; 45% Cox HRs and PH check; 10% plot
R03.02.01.01-T04 Forecast 12 weeks of demand and beat a seasonal-naive baseline
Original instruction
data/demand_weekly.csv has 4 years of weekly demand for 30 product families with holidays. Produce 12-week-ahead forecasts per family in output/forecast.csv. The last 12 weeks are held out by the grader.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · statsmodels · pandas
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader computes weighted MAPE vs hidden actuals: pass at ≤0.85× the seasonal-naive WMAPE; partial credit linearly up to 1.0×. All 30×12 values present and non-negative.
Proposed partial credit
10% complete output; 90% WMAPE vs baseline (linear)
R03.02.01.01-T05 Detect and explain a Simpson's paradox in referral conversion
Original instruction
Marketing claims referral leads convert worse than paid leads (data/leads_2025.csv). Determine whether this holds within each segment, identify the confounder, and report pooled and stratified conversion rates plus a Mantel-Haenszel odds ratio in output/answer.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · pandas · statsmodels
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader checks: confounder named = 'company_size' (exact field), stratified rates ±0.001, MH OR ±0.01, and conclusion field 'reverses' true. Rates computed from data, not stated.
Proposed partial credit
25% confounder; 35% stratified rates; 25% MH odds ratio; 15% conclusion
R03.02.01.01-T06 Produce a descriptive statistics table to a reporting spec
Original instruction
From data/patients.csv, produce the 'Table 1' in spec/table1.md: N, mean (SD) or median [IQR] per variable per arm, counts (%) for categorical variables, and p-values with the tests the spec names. Save output/table1.csv.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- R · pandas
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Cell-by-cell comparison with gold: formatted strings exact (rounding per spec); p-values ±0.001.
Proposed partial credit
70% summary cells; 30% p-values
R03.02.01.01-T07 Handle missing data with multiple imputation and compare estimates
Original instruction
data/wellbeing.csv has 28% missing income and 12% missing sleep, not missing at random by age. Fit the regression in spec/model.md with complete cases and with multiple imputation (m=20, chained equations, seed 7), pool with Rubin's rules, and report both in output/mi.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- R · mice
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Gold from mice with the same seed: pooled coefficients ±0.01, SEs ±5%, fraction of missing information ±0.02; complete-case estimates exact. Hidden simulation check: the MI estimate is closer to the known truth than complete-case.
Proposed partial credit
25% complete-case fit; 45% pooled MI estimates/SEs; 15% FMI; 15% truth recovery
R03.02.01.01-T08 Estimate a policy effect with difference-in-differences
Original instruction
data/county_panel.csv covers 2015–2024 for 400 counties; 60 adopted a wage policy in 2019 (staggered). Estimate the employment effect with a two-way FE DiD and the Callaway–Sant'Anna estimator, run an event-study pre-trend test, and report both in output/did.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- R · did · fixest
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Gold: TWFE coefficient ±0.002; CS overall ATT ±0.003 with SE ±10%; pre-trend joint test p ±0.02; event-study coefficients for −3..+3 ±0.003.
Proposed partial credit
20% TWFE; 40% CS ATT/SE; 20% event study; 20% pre-trend test
R03.02.01.01-T09 Compute sample size for a two-proportion test
Original instruction
Product wants to detect a lift from 4.2% to 4.6% conversion with 80% power at α=0.05 two-sided, with 3 variants vs control and Bonferroni correction. Compute per-arm sample size and days needed at 18k eligible users/day, writing output/power.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Python · statsmodels
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Gold computed with statsmodels NormalIndPower (Cohen's h): per-arm n ±1%, total n and days (ceil) exact.
Proposed partial credit
60% per-arm n; 20% correction applied; 20% days
R03.02.01.01-T10 Segment customers with k-means under a fixed preprocessing spec
Original instruction
Follow spec/segmentation.md on data/customers_rfm.csv: log-transform monetary, standardize, choose k in 2–10 by silhouette, fit k-means (seed 42, n_init 20), and write output/segments.csv with labels and output/profile.csv with per-segment means.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · scikit-learn
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader: chosen k equals gold; ARI vs gold labels ≥0.98; profile means ±0.01. Notebook reruns deterministically.
Proposed partial credit
30% correct k; 50% label agreement; 20% profiles
R03.02.01.01-T11 Run a repeated-measures ANOVA in SPSS
Original instruction
In SPSS Statistics, open data/reaction_times.sav (24 participants × 4 conditions) and run a repeated-measures ANOVA with Mauchly's test, Greenhouse–Geisser correction and Bonferroni post-hoc comparisons. Save the output viewer file to output/rm_anova.spv and the key statistics to output/anova.json.
- Platforms
mac· macOSwin· Windows- Applications
- SPSS Statistics
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Grader: anova.json F, df (GG-corrected), p, partial eta² ±0.01 vs gold; post-hoc pairs and adjusted p ±0.005. The .spv must contain the Mauchly and Tests of Within-Subjects Effects tables (parsed via SPSS output export).
Proposed partial credit
40% corrected F/df/p; 15% Mauchly; 30% post-hoc; 15% .spv saved
R03.02.01.01-T12 Fit a panel regression with clustered errors in Stata
Original instruction
Using data/firms.dta, write output/analysis.do that fits a firm fixed-effects regression of log productivity on R&D intensity and controls with year dummies and SEs clustered by industry, then exports the table to output/table.csv via esttab.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Stata
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs the .do in batch Stata: coefficients ±1e-4 and clustered SEs ±1% of gold; N and within-R² exact; table.csv columns per spec.
Proposed partial credit
50% coefficients; 30% clustered SEs; 20% exported table
R03.02.01.01-T13 Repair a broken analysis notebook so it reproduces its figures
Original instruction
notebooks/q3_review.ipynb fails at cell 14 (renamed columns, deprecated pandas API, a hidden-state variable defined in a deleted cell). Fix it so 'Restart & Run All' completes and reproduces the four numbers quoted in the markdown summary.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · pandas
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 3 / 5
Proposed verifier
Grader executes the notebook in a fresh kernel with nbclient: no errors; the 4 summary numbers recomputed by grader equal those printed in the notebook outputs and match the markdown values ±0.01.
Proposed partial credit
40% runs end to end; 40% numbers reproduce; 20% no hidden-state dependencies (cell order randomized test)
R03.02.01.01-T14 Attribute a week-over-week conversion drop to segments
Original instruction
Conversion fell from 3.9% to 3.4% last week (data/funnel_daily.parquet). Decompose the change into mix effect and rate effect across device × channel × country, and name the top 3 contributing segments with their contributions in output/drop.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · pandas
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Gold: total change −0.5 pp decomposed into mix and rate (±0.01 pp); top 3 segments exact and in order; contributions ±0.02 pp; contributions sum to the total.
Proposed partial credit
30% mix/rate split; 50% top segments; 20% sums consistent
R03.02.01.01-T15 Fit a hierarchical Bayesian model of store conversion
Original instruction
Fit a hierarchical logistic model in PyMC for data/store_visits.csv (210 stores, many with few visits): store intercepts partially pooled within region. Report posterior mean and 94% HDI of each region's conversion, and the probability that store S-117 beats its region mean, in output/posterior.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- PyMC · ArviZ · Jupyter
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Grader checks convergence (R-hat ≤1.01, ESS ≥400) from the saved trace, region means ±0.002 and HDI bounds ±0.004 of the reference posterior, and P(S-117 > region) ±0.03.
Proposed partial credit
20% convergence; 40% region posteriors; 20% HDIs; 20% S-117 probability
R03.02.01.01-T16 Estimate a treatment effect with propensity score matching
Original instruction
data/training_program.csv records which employees took a training program and their later promotion. Estimate the ATT on promotion using 1:1 nearest-neighbor propensity matching with caliper 0.2 SD of the logit, report balance (standardized mean differences) before and after, and the ATT with bootstrap CI.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- R · MatchIt
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Gold via MatchIt with the same spec: matched N exact, SMDs after matching ±0.01 (all <0.1), ATT ±0.005, CI ±0.01. Hidden semi-synthetic version with known effect must be recovered ±0.02.
Proposed partial credit
20% matched sample; 25% balance table; 35% ATT/CI; 20% semi-synthetic recovery
R03.02.01.01-T17 Fit a mixed-effects model for repeated measures in R
Original instruction
data/sleep_trial.csv has nightly sleep scores for 180 participants over 6 weeks in 3 arms. Fit an lme4 model with random intercepts and slopes by participant, test the arm × week interaction with a likelihood-ratio test, and report fixed effects and the LRT in output/mixed.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- R · lme4
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Gold: fixed-effect estimates ±0.005, SEs ±3%, LRT chi-square ±0.1 and p ±0.005; random-effect variances ±5%. Model must be fit with REML false for the LRT.
Proposed partial credit
40% fixed effects; 20% random effects; 30% LRT; 10% correct REML setting
R03.02.01.01-T18 Compute bootstrap confidence intervals for ratio metrics
Original instruction
For data/sessions.parquet, compute 95% BCa bootstrap CIs (10,000 resamples, seed 11, user-level clusters) for median session length and for revenue per session in each of 2 app versions, and the CI of their difference. Write output/boot.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Python · SciPy
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Gold computed with the same seed and cluster resampling: point estimates exact, CI bounds ±1.5% (Monte Carlo tolerance). Session-level (non-clustered) resampling yields narrower CIs and fails.
Proposed partial credit
30% point estimates; 50% cluster BCa CIs; 20% difference CI
R03.02.01.01-T19 Control false discoveries across 200 experiment metrics
Original instruction
data/metric_tests.csv lists 200 metrics with control/treatment sums and counts. Compute per-metric Welch t-test p-values, apply Benjamini–Hochberg at q=0.05 and Holm at 0.05, and list discoveries under each in output/fdr.json.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Python · statsmodels
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Gold p-values ±1e-4; discovery sets exact under both procedures.
Proposed partial credit
40% p-values; 35% BH set; 25% Holm set
R03.02.01.01-T20 Answer ten graded questions in a full EDA of a new dataset
Original instruction
data/city_bikes/ has 2 years of trip, station and weather data. In a Jupyter notebook, clean it per docs/notes.md and answer the 10 questions in questions.md (e.g., weather-adjusted ridership trend, station imbalance, effect of rain), writing answers to output/answers.json with the specified units.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · pandas · statsmodels
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Each answer compared to gold with per-question tolerance; the notebook must reproduce answers.json from raw data in ≤15 min. One question is ill-posed by design and must be answered 'insufficient data' with reason.
Proposed partial credit
72% nine answerable questions (8 pts each); 13% notebook reproduces; 15% ill-posed question flagged with reason
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.