Research reproduction & experiments
Reproduce a paper's result or run a research experiment and report it.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- PyTorch · Jupyter · Docker
- Class GUI-need label
CL· CLI-sufficient
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R03.03.01.02-T01 Reproduce a label-smoothing results table on CIFAR-10
Original instruction
papers/label_smoothing.pdf Table 2 reports ResNet-20 CIFAR-10 test error and ECE for smoothing ε ∈ {0, 0.05, 0.1, 0.2} (3 seeds). Reproduce the table from scratch with the paper's training recipe on 1 GPU and write output/table2.csv (eps, error_mean, error_std, ece_mean).
- Platforms
lin· Linux- Applications
- PyTorch · Docker
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader reruns the agent's run script for one ε and seed (must match logged result ±0.2%), then compares table2.csv to the paper: error within ±0.5 pp and ECE within ±0.5 pp for all ε; trend (ECE minimum at ε=0.1) preserved.
Proposed partial credit
15% recipe matches paper (config check); 50% error values; 25% ECE values; 10% spot rerun consistent
R03.03.01.02-T02 Reproduce a warmup ablation figure with logged curves
Original instruction
Reproduce Figure 4 of papers/warmup.pdf: validation loss curves of a 6-layer transformer on data/wikitext-2 with 0, 500 and 2,000 warmup steps (same peak LR). Save output/curves.csv (step, setting, val_loss) and output/fig4.png.
- Platforms
lin· Linux- Applications
- PyTorch · Jupyter
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader checks final val loss per setting within ±0.05 of the paper's reported values and the ordering (no-warmup diverges or ends highest). Curves must come from the agent's logs (spot-rerun of 300 steps matches ±0.02).
Proposed partial credit
50% final losses; 20% ordering/qualitative claim; 20% spot rerun; 10% figure
R03.03.01.02-T03 Complete missing pieces of a paper's starter code and run it
Original instruction
repo/contrastive_ts/ is the official code for papers/ts2vec_like.pdf with the hierarchical contrastive loss and the timestamp masking removed. Implement both from Section 3, then reproduce the UCR average accuracy on the 10 datasets listed in configs/ucr10.yaml.
- Platforms
lin· Linux- Applications
- PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden unit tests compare the loss to reference values on fixed tensors (±1e-5) and masking statistics. Average accuracy over the 10 datasets within ±1.0 pp of the paper's subset value; per-dataset results in output/ucr10.csv.
Proposed partial credit
35% loss unit tests; 15% masking tests; 50% accuracy reproduction
R03.03.01.02-T04 Run an ablation with three seeds and significance tests
Original instruction
For the model in repo/tabnet_lite on data/adult, run the 4 ablations in spec/ablations.md (remove attention, remove sparsity loss, single step, no feature reuse) with 3 seeds each, and report mean ± std accuracy and paired t-test p-values vs the full model in output/ablation.csv.
- Platforms
lin· Linux- Applications
- PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader reruns one random (ablation, seed) pair: matches logged accuracy ±0.1 pp. Means/stds recomputed from logs equal the CSV; p-values recomputed match ±0.005. All 15 runs present.
Proposed partial credit
30% all runs logged; 30% table consistent with logs; 20% p-values; 20% spot rerun
R03.03.01.02-T05 Fit a scaling law and predict a held-out model size
Original instruction
Train 6 GPT-style models (1M–30M params) on data/tinystories with the compute-optimal token budget in spec/scaling.md, fit L(N) = a·N^-α + c, and predict the loss of a 60M model. Write output/scaling.json with the fit and prediction.
- Platforms
lin· Linux- Applications
- PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader trains the 60M model with the same recipe (pre-computed reference): prediction within ±5% of actual loss. Fit parameters recomputed from the agent's logged losses ±2%. Each of the 6 runs must have a training log consistent with spec token counts.
Proposed partial credit
30% six runs per spec; 25% fit reproducible; 45% 60M prediction accuracy
R03.03.01.02-T06 Replicate a published regression table from replication data
Original instruction
papers/min_wage_replication.pdf Table 3 reports 4 regressions. Using the authors' replication files in data/replication/ (Stata .dta and a partial R script), reproduce all coefficients and standard errors in R and write output/table3.csv.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- R · haven · fixest
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader compares 4 columns × 6 coefficients and SEs to the published table: coefficients ±0.001, SEs ±0.002 (clustered as in paper). Script output/replicate.R must reproduce the CSV.
Proposed partial credit
60% coefficients; 30% SEs; 10% script reproduces
R03.03.01.02-T07 Containerize a research repo for deterministic reproduction
Original instruction
repo/graph_pool/ only runs on the author's machine. Write a Dockerfile and run.sh that pin CUDA, PyTorch and package versions, download nothing at runtime (datasets vendored in data/), and reproduce the README's reported accuracy on the PROTEINS dataset deterministically.
- Platforms
lin· Linux- Applications
- Docker · PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader builds the image offline from the context (network disabled during run), executes run.sh twice: identical metrics both times and within ±0.5 pp of README value. Image ≤8 GB.
Proposed partial credit
30% image builds offline; 30% deterministic reruns; 30% accuracy matches README; 10% size
R03.03.01.02-T08 Test whether a paper's headline claim survives seed variance
Original instruction
papers/opt_claim.pdf claims optimizer X beats AdamW on 3 small benchmarks. Using its released code, rerun both optimizers with 5 seeds each per benchmark, and report in output/verdict.json whether the claim holds per benchmark at α=0.05 (Welch test), with effect sizes.
- Platforms
lin· Linux- Applications
- PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Gold verdicts from reference runs (claim holds on 1 of 3). Grader checks verdicts exact, effect sizes ±0.1, and that logs contain 30 runs with the stated seeds; one spot rerun must match.
Proposed partial credit
45% verdicts; 20% effect sizes; 25% complete runs; 10% spot rerun
R03.03.01.02-T09 Reproduce parameter estimates from a computational biology paper
Original instruction
papers/predator_prey.pdf fits a Lotka–Volterra model with harvesting to data/lynx_hare.csv and reports parameters and 95% CIs in Table 1. Reproduce the fit (same objective and optimizer settings), parameters and profile-likelihood CIs in a Jupyter notebook.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jupyter · SciPy
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Parameters within ±1% of the paper; CI bounds within ±3%; notebook reruns end to end and reproduces output/table1.csv.
Proposed partial credit
50% point estimates; 35% CIs; 15% notebook reproduces
R03.03.01.02-T10 Port TensorFlow 1 research code to PyTorch with matching outputs
Original instruction
repo/tf1_capsnet/ implements a capsule network in TensorFlow 1.15 with released checkpoints. Port the model and routing to PyTorch, convert the checkpoint, and match the TF outputs on data/mnist_test, then fine-tune to the paper's accuracy.
- Platforms
lin· Linux- Applications
- PyTorch · TensorFlow
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader feeds 1,000 test images to both: logits match ±1e-4 (max abs). Converted model accuracy equals TF accuracy exactly; agent's PyTorch training from scratch reaches ≥99.4%.
Proposed partial credit
50% output parity; 20% converted accuracy; 30% from-scratch training target
R03.03.01.02-T11 Implement a loss from pseudocode and validate it against reference values
Original instruction
Implement the preference loss in papers/pref_loss.pdf (Algorithm 1, including the reference-model log-ratio and label smoothing) in src/loss.py, then use it to train the small model in repo/pref_tiny for 1 epoch and report win-rate vs the SFT model with the provided judge script.
- Platforms
lin· Linux- Applications
- PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden unit tests compare loss and gradients to reference values on 20 fixed batches (±1e-5). Trained model win-rate ≥0.58 against SFT on the 500 held-out prompts using the provided deterministic reward model.
Proposed partial credit
50% loss/gradient tests; 50% win-rate target
R03.03.01.02-T12 Run a hyperparameter sensitivity study and write it up
Original instruction
For repo/vit_small on data/tiny_imagenet_100, sweep learning rate (5 values) × weight decay (4 values) with 2 seeds, write output/sweep.csv, a heatmap output/sensitivity.png, and output/report.md stating the optimal region and how sensitive accuracy is.
- Platforms
lin· Linux- Applications
- PyTorch · Jupyter
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
sweep.csv must contain 40 runs consistent with logs; grader spot-reruns 2 configs (±0.3 pp). Report's stated best (lr, wd) must equal the CSV argmax. LLM rubric (20%) for whether conclusions follow the data.
Proposed partial credit
35% complete sweep; 20% spot reruns; 25% report facts consistent; 20% rubric
R03.03.01.02-T13 Reproduce reported zero-shot benchmark numbers with an eval harness
Original instruction
The model card of models/pythia_410m_ft reports 5 zero-shot scores (ARC-e, HellaSwag, PIQA, WinoGrande, LAMBADA). Reproduce them with lm-evaluation-harness (offline, datasets in data/), writing output/scores.json, and explain any gap larger than 1 point.
- Platforms
lin· Linux- Applications
- lm-evaluation-harness · PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader runs the same harness config: scores equal grader's ±0.2. Gaps vs model card: the seeded mismatch (normalized vs raw accuracy on HellaSwag) must be named in output/notes.md.
Proposed partial credit
60% scores reproduced; 25% gap explained; 15% config logged
R03.03.01.02-T14 Find why a paper's reported results cannot be reproduced
Original instruction
repo/fewshot_proto/ claims 78.4% 5-way 1-shot accuracy on miniImageNet but you get ~62%. Investigate the code and evaluation protocol, find the cause of the inflated number, correct it, and report honest results with 95% CIs over 2,000 episodes in output/findings.md and output/results.json.
- Platforms
lin· Linux- Applications
- PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 5 / 5
Proposed verifier
Seeded cause: test classes overlap with training classes in the episode sampler. Grader checks findings.md names it (keyword + file/line), the fixed sampler passes a disjointness test, and results.json accuracy ±0.5 pp of the reference honest number with CI ±0.2.
Proposed partial credit
40% cause identified with location; 30% fix verified; 30% honest results
R03.03.01.02-T15 Replicate PPO sample-efficiency curves on classic control tasks
Original instruction
Replicate the PPO curves for CartPole-v1 and Acrobot-v1 in papers/ppo_baselines.pdf Figure 2 (5 seeds, 200k steps) using your own PyTorch implementation, and save output/curves.csv and output/fig2.png.
- Platforms
lin· Linux- Applications
- PyTorch · Gymnasium
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader checks final mean return within the paper's shaded band for both tasks and steps-to-threshold (CartPole 475) within ±25%; reruns one seed for 50k steps to confirm logs.
Proposed partial credit
40% final returns; 30% sample efficiency; 20% spot rerun; 10% figure
R03.03.01.02-T16 Execute a pre-registered experiment protocol end to end
Original instruction
spec/prereg.md pre-registers an experiment: does mixup improve calibration of a CNN on data/fashion_mnist? Follow it exactly (power analysis for seed count, primary metric ECE, Holm correction for 2 secondary metrics), run it, and write output/results.md with deviations declared.
- Platforms
lin· Linux- Applications
- PyTorch · SciPy
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader checks seed count equals the power-analysis result, metrics computed as specified (recompute from logs), corrected p-values exact, and conclusion matches data. Undeclared deviations (e.g., changed epochs) detected from configs fail that milestone.
Proposed partial credit
20% power analysis; 40% runs and metrics per protocol; 20% corrections; 20% no undeclared deviations
R03.03.01.02-T17 Reproduce a competition solution's cross-validation score
Original instruction
docs/solution_writeup.md describes a top solution for a tabular competition (features, model blend, CV scheme). Using data/competition/, implement it and reproduce its reported 5-fold CV score of 0.7932 AUC within ±0.002.
- Platforms
lin· Linux- Applications
- LightGBM · CatBoost · pandas
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader reruns the agent's pipeline: CV AUC within ±0.002 of 0.7932; fold assignment follows writeup (GroupKFold by user). Score on hidden private split ≥0.790.
Proposed partial credit
50% CV score; 20% correct CV scheme; 30% private split score
R03.03.01.02-T18 Turn raw experiment logs into paper-ready tables and figures
Original instruction
logs/ has 48 JSONL training logs (4 methods × 4 datasets × 3 seeds). Produce output/table.tex (mean ± std, best per column bold, booktabs) and output/fig.pdf (learning curves with shaded std) exactly per spec/figure_spec.md.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Python · matplotlib · LaTeX
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP03· Visual layout- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader parses table.tex: values equal recomputed means/stds to 2 decimals; bold cells correct; compiles with pdflatex. fig.pdf checked for 4 panels and legend entries (PDF text extraction).
Proposed partial credit
50% table values; 20% bolding and format; 30% figure spec
R03.03.01.02-T19 Replicate a paper's core contribution against a rubric
Original instruction
Replicate the main contribution of papers/icml_selected.pdf from scratch (no released code): implement the method, run the 3 core experiments listed in rubric/scope.md on 1 GPU within 12 hours, and write output/replication.md comparing your results with the paper.
- Platforms
lin· Linux- Applications
- PyTorch · Jupyter
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Hierarchical rubric (subset of 120 leaves): code-development leaves checked by tests/inspection, execution leaves by logs, result leaves by numeric tolerance vs paper. LLM judge only for code-development leaves lacking tests, stated in the rubric.
Proposed partial credit
30% code-development leaves; 30% execution leaves; 40% result-match leaves
R03.03.01.02-T20 Reproduce grokking on modular arithmetic
Original instruction
Reproduce the grokking phenomenon: train a 2-layer transformer on (a + b) mod 97 with 30% training data and weight decay 1.0 as in papers/grokking.pdf, logging train/val accuracy. Report the step where validation accuracy first exceeds 99% in output/grok.json.
- Platforms
lin· Linux- Applications
- PyTorch
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP07· Simulation result- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader checks logs: train acc ≥99% well before val (gap ≥5k steps), val acc ≥99% by 50k steps; reported step equals the logs; with weight decay 0 (agent must also run) val stays <50% at 50k.
Proposed partial credit
40% grokking observed; 20% reported step consistent; 30% no-WD control; 10% plots
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.