R03.03.01.01 · TASK CLASS

End-to-end ML engineering

Train and tune a model to a target metric on a held-out set.

20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.

Typical applications: PyTorch · scikit-learn · Kaggle data
Class GUI-need label: CL · CLI-sufficient

Task proposals · 20

Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.

R03.03.01.01-T01 Train a churn classifier to a target AUC

Original instruction

data/churn/train.csv (48k customers, 40 features, label churned_90d) and test_features.csv are provided. Build a model and write predict.py that reads a features CSV and writes probabilities to a given path; also write output/test_preds.csv for the test set.

Platforms: lin · Linux mac · macOS win · Windows
Applications: scikit-learn · LightGBM
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Grader scores test_preds.csv against hidden labels: ROC-AUC ≥0.86 passes (linear credit from 0.80). predict.py is rerun on a hidden second test split and must also reach ≥0.85, guarding against label leakage or overfitting to test.

Proposed partial credit

60% AUC on test; 30% AUC on second split; 10% predict.py interface

R03.03.01.01-T02 Reach 75% CIFAR-10 accuracy on CPU in 30 minutes

Original instruction

Train a CNN with PyTorch on CIFAR-10 (data/cifar10/) using CPU only (8 cores) with a 30-minute wall-clock budget, and save model.pt plus eval.py that reports test accuracy.

Platforms: lin · Linux
Applications: PyTorch
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader retrains from the agent's train.py in a fresh container with the same CPU limit, kills at 30 min, then evaluates on the test set: accuracy ≥75% passes (linear from 60%). No pretrained weights may be downloaded (network blocked).

Proposed partial credit

70% accuracy target; 20% finishes within budget; 10% eval.py correct

R03.03.01.01-T03 Fine-tune DistilBERT to classify support tickets

Original instruction

data/tickets/ has 22k labeled tickets in 14 categories with heavy imbalance. Fine-tune distilbert-base-uncased (weights in models/) on 1 GPU to macro-F1 ≥0.82 on the hidden test set; deliver predict.py and the saved model.

Platforms: lin · Linux
Applications: PyTorch · Hugging Face Transformers
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader runs predict.py on hidden test: macro-F1 ≥0.82 passes (linear from 0.70); per-class recall ≥0.5 for all 14 classes; inference on 2k tickets ≤60 s on the grader GPU.

Proposed partial credit

60% macro-F1; 25% per-class recall floor; 15% inference time

R03.03.01.01-T04 Forecast store-item sales with gradient boosting

Original instruction

data/store_sales/ has 3 years of daily sales for 50 stores × 40 items plus promotions and holidays. Forecast the next 28 days for every store-item and write output/forecast.csv; deliver the training pipeline as train.py.

Platforms: lin · Linux mac · macOS win · Windows
Applications: LightGBM · pandas
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader computes sMAPE vs hidden actuals: ≤12.5% passes (linear from 16%, seasonal-naive ≈17%). train.py rerun on data shifted by 28 days must produce a forecast within 10% sMAPE of the agent's.

Proposed partial credit

70% sMAPE target; 20% pipeline reproduces; 10% complete output

R03.03.01.01-T05 Fine-tune a detector on a small defect dataset

Original instruction

data/pcb_defects/ has 1,200 annotated images (6 defect classes, COCO format). Fine-tune an object detector (pretrained YOLO weights in models/) to mAP@0.5 ≥0.60 on the hidden test split and deliver predict.py writing COCO-format detections.

Platforms: lin · Linux
Applications: PyTorch · Ultralytics YOLO
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Grader runs predict.py on hidden test images and scores with pycocotools: mAP@0.5 ≥0.60 passes (linear from 0.40); mAP@0.5:0.95 reported; small-defect class AP ≥0.35.

Proposed partial credit

60% mAP@0.5; 25% small-class AP; 15% valid COCO output

R03.03.01.01-T06 Build a recommender that beats popularity by 40% Recall@20

Original instruction

data/interactions.parquet holds 6M implicit user-item events (8 weeks). Train a recommender (ALS, two-tower or similar) and write output/recs.parquet with 20 items for each of the 50k evaluation users; the last week is hidden.

Platforms: lin · Linux
Applications: PyTorch · implicit
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader computes Recall@20 on the hidden week: ≥1.4× the popularity baseline passes (linear from 1.0×). Recommending already-seen items counts against recall per spec; coverage ≥5% of catalog required.

Proposed partial credit

70% recall lift; 15% coverage; 15% valid output for all users

R03.03.01.01-T07 Find and remove target leakage from a training pipeline

Original instruction

pipelines/readmit/ reports AUC 0.97 for 30-day hospital readmission, but production AUC is 0.68. Find the leaking features and preprocessing steps, fix the pipeline, and report the leaks in output/leaks.md and the honest validation AUC in output/metrics.json.

Platforms: lin · Linux mac · macOS win · Windows
Applications: scikit-learn · pandas
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: hidden · Hidden/stale state
Proposed difficulty: 3 / 5

Proposed verifier

Grader checks leaks.md names the 3 seeded leaks (post-discharge feature, scaler fit on all data, patient overlap across folds). Fixed pipeline's reported AUC within ±0.02 of grader's grouped temporal CV and ≥0.72 on the hidden future cohort.

Proposed partial credit

45% leaks identified (15 pts each); 25% honest metric matches; 30% future-cohort AUC

R03.03.01.01-T08 Tune a gradient-boosting model with Optuna under a trial budget

Original instruction

Tune an XGBoost regressor on data/energy/ with Optuna in at most 60 trials (5-fold time-series CV) and deliver output/best_params.json, output/study.db and the retrained model. Target RMSE ≤ 18.0 on the hidden test year.

Platforms: lin · Linux mac · macOS win · Windows
Applications: XGBoost · Optuna
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader opens study.db: ≤60 completed trials, CV scheme TimeSeriesSplit; retrains with best_params and scores hidden test: RMSE ≤18.0 passes (linear from 22.0).

Proposed partial credit

60% test RMSE; 20% budget respected; 20% CV scheme correct

R03.03.01.01-T09 Detect fraud at high recall with a calibrated threshold

Original instruction

data/transactions/ is 2.4M card transactions with 0.17% fraud. Train a model, calibrate probabilities, and choose a threshold that gives recall ≥0.80 on the hidden test month; report the threshold and expected precision in output/threshold.json and deliver predict.py.

Platforms: lin · Linux
Applications: LightGBM · scikit-learn
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Grader on hidden month: recall at the chosen threshold ≥0.80; precision at that point ≥0.35 passes (linear from 0.20); Brier score ≤ baseline×0.9; stated precision within ±0.05 of actual.

Proposed partial credit

35% recall met; 35% precision; 15% calibration; 15% precision estimate accurate

R03.03.01.01-T10 Segment buildings in satellite tiles to a target mIoU

Original instruction

data/buildings/ has 4k 512×512 RGB tiles with binary masks. Train a segmentation model (U-Net or similar) on 1 GPU to mIoU ≥0.72 on the hidden test tiles; deliver predict.py writing PNG masks.

Platforms: lin · Linux
Applications: PyTorch · segmentation_models_pytorch
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Grader runs predict.py on hidden tiles: mIoU ≥0.72 passes (linear from 0.60); boundary F1 reported; masks must be the input size and binary.

Proposed partial credit

70% mIoU; 15% boundary F1 ≥0.6; 15% output format

R03.03.01.01-T11 Distill a classifier into a CPU model under 10 ms p95

Original instruction

A fine-tuned BERT-base intent classifier (models/teacher/) gets 94.1% accuracy but runs at 45 ms on CPU. Distill a student that reaches ≥93.0% accuracy on the hidden test set with p95 latency ≤10 ms per query on 1 CPU core, exported to ONNX.

Platforms: lin · Linux
Applications: PyTorch · ONNX Runtime
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 5 / 5

Proposed verifier

Grader loads the ONNX student in ONNX Runtime with 1 thread: p95 latency over 2,000 single queries ≤10 ms; accuracy ≥93.0% (linear from 90%). Teacher must not be called at inference (graph size check).

Proposed partial credit

50% accuracy; 40% latency; 10% valid ONNX export

R03.03.01.01-T12 Predict house prices with heavy missingness

Original instruction

data/houses/train.csv (1,460 rows, 79 features, many missing) and test.csv are given. Build a regression pipeline and write output/preds.csv with SalePrice for the test set.

Platforms: lin · Linux mac · macOS win · Windows
Applications: scikit-learn · pandas
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Grader computes RMSLE vs hidden prices: ≤0.125 passes (linear from 0.16). Pipeline must refit end to end from train.py in ≤10 min.

Proposed partial credit

80% RMSLE; 20% reproducible pipeline

R03.03.01.01-T13 Classify spoken commands from audio

Original instruction

data/speech_commands/ has 1-second WAV clips for 12 classes (10 words, silence, unknown). Train a model to ≥92% accuracy on the hidden test set using log-mel features or a small CNN, and deliver predict.py.

Platforms: lin · Linux
Applications: PyTorch · torchaudio
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader runs predict.py on hidden clips: accuracy ≥92% passes (linear from 80%); 'unknown' recall ≥0.8; inference ≤5 ms per clip on CPU.

Proposed partial credit

70% accuracy; 15% unknown recall; 15% latency

R03.03.01.01-T14 Quantize a model to INT8 with under 1% accuracy loss

Original instruction

Export the ResNet-50 in models/resnet_food101.pt (Food-101, top-1 84.6%) to ONNX and quantize it to INT8 with static calibration on data/calib/ so that top-1 drops by ≤1.0 point and CPU throughput at least doubles.

Platforms: lin · Linux
Applications: PyTorch · ONNX Runtime
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader evaluates the INT8 ONNX on the hidden test set: top-1 ≥83.6% (linear from 82%); throughput on 8 cores ≥2× the FP32 ONNX baseline; model contains QLinear ops.

Proposed partial credit

50% accuracy retained; 35% speedup; 15% truly quantized graph

R03.03.01.01-T15 Fix NaN losses and get a transformer to train to target

Original instruction

train_lm.py in fixtures/tiny_lm diverges to NaN after ~800 steps on data/wiki_small. Diagnose (learning-rate schedule, missing gradient clipping, fp16 overflow in attention, bad init), fix it, and train to validation perplexity ≤38 within 2 GPU-hours.

Platforms: lin · Linux
Applications: PyTorch
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: hidden · Hidden/stale state
Proposed difficulty: 4 / 5

Proposed verifier

Grader reruns training from the fixed code with 2 seeds: no NaN/inf in logged loss; validation perplexity ≤38 (linear from 50) for both seeds; output/diagnosis.md names the root causes matching the seeded 2 bugs.

Proposed partial credit

50% perplexity target; 25% stable on both seeds; 25% root causes identified

R03.03.01.01-T16 Train a named-entity recognizer for invoices

Original instruction

data/invoice_ner/ has 6k tokenized invoice texts with BIO tags for 9 entity types. Train a model to entity-level F1 ≥0.88 on the hidden test set (seqeval) and deliver predict.py producing BIO tags.

Platforms: lin · Linux
Applications: PyTorch · Hugging Face Transformers
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader runs seqeval on hidden test: micro F1 ≥0.88 passes (linear from 0.75); every entity type F1 ≥0.70; output token alignment exact.

Proposed partial credit

60% micro F1; 25% per-type floor; 15% alignment

R03.03.01.01-T17 Categorize products from images and titles together

Original instruction

data/products/ has 60k items with an image, title and description across 48 leaf categories in a 3-level taxonomy. Train a multimodal classifier (pretrained encoders in models/) that reaches leaf accuracy ≥0.85 and level-1 accuracy ≥0.97 on the hidden test set.

Platforms: lin · Linux
Applications: PyTorch · OpenCLIP
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Grader runs predict.py on hidden test: leaf accuracy ≥0.85 (linear from 0.70), level-1 ≥0.97; predictions must be valid taxonomy paths. Text-only or image-only ablation reported in output/ablation.json must match grader reruns ±0.02.

Proposed partial credit

50% leaf accuracy; 20% level-1 accuracy; 15% valid paths; 15% ablation report

R03.03.01.01-T18 Adapt a defect classifier to a new factory with 200 labels

Original instruction

A classifier trained on factory A (models/factory_a.pt, data/factory_a/) drops to 61% accuracy on factory B's images. Using the unlabeled factory B set and only 200 labeled B images, raise accuracy on B's hidden test set to ≥85% without falling below 88% on A.

Platforms: lin · Linux
Applications: PyTorch
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Grader evaluates on both hidden test sets: B accuracy ≥85% (linear from 70%); A accuracy ≥88%. Training script must use only the 200 labeled B images (label file hash checked).

Proposed partial credit

60% B accuracy; 25% A retained; 15% label budget respected

R03.03.01.01-T19 Make a training pipeline reproducible across reruns

Original instruction

fixtures/credit_pipeline gives different validation AUCs on every run (0.781–0.804). Make it deterministic: seed everything, pin data ordering and splits, fix nondeterministic ops, and add a config file, so three reruns produce identical metrics and model hashes.

Platforms: lin · Linux mac · macOS win · Windows
Applications: PyTorch · scikit-learn
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Grader runs the pipeline 3× in fresh processes: identical metric values and identical saved model weights (hash). Validation AUC ≥0.79.

Proposed partial credit

60% identical reruns; 20% weights identical; 20% AUC floor

R03.03.01.01-T20 LoRA-tune a small LLM for text-to-SQL

Original instruction

Fine-tune the 1.5B model in models/base_llm with LoRA on data/sql_pairs.jsonl (28k NL-SQL pairs over 40 SQLite schemas) using 1 GPU, and deliver generate.py that outputs SQL for a question and schema. Target ≥62% execution accuracy on the hidden test schemas.

Platforms: lin · Linux
Applications: PyTorch · PEFT · Hugging Face Transformers
Interfaces: term · Terminal
GUI need · task setting: CL · CLI-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP06 · Code/engineering
Conditions: long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Grader runs generate.py on 500 hidden questions over 8 unseen schemas and executes against SQLite: execution accuracy ≥62% passes (linear from base model ≈38%). Generation ≤2 s per question.

Proposed partial credit

75% execution accuracy; 15% latency; 10% valid SQL rate ≥0.95

Mapped research references

Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.

End-to-end ML engineering

Task proposals · 20

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Mapped research references

In this workflow family