R03.03.02.01 · TASK CLASS

Experiment tracking & model registry

Compare runs, select and register models, and read curves in tracking UIs.

20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.

Typical applications: Weights & Biases · MLflow · TensorBoard
Class GUI-need label: G2 · GUI-essential

Task proposals · 20

Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.

R03.03.02.01-T01 Pick and tag the best qualifying run in a W&B project

Original instruction

In the W&B sandbox project 'resnet-ablations' (48 runs), find the run with the highest final val/acc among runs with batch_size 64 whose final train/acc − val/acc gap is under 2 points. Tag it 'candidate' and add the note 'selected for eval' to it.

Platforms: web · Any OS (web)
Applications: Weights & Biases
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click vis · Visual judgment
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader reads run tags and notes with an admin key: exactly one run tagged 'candidate', equal to gold; note text exact. Agent session has no API key and the wandb SDK is not installed; tag edits must come from the web UI (audit log user agent).

Proposed partial credit

60% correct run; 25% tag only on that run; 15% note

R03.03.02.01-T02 Register the best MLflow run as a new model version in Staging

Original instruction

In the local MLflow UI, compare runs in experiment 'churn-xgb' and register the model artifact of the run with the highest val_auc as model 'churn-model' (new version) with description 'val_auc=<value>', then set alias 'staging' on it.

Platforms: web · Any OS (web)
Applications: MLflow
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Grader queries the registry: new version source run_id equals the best run; description contains the exact AUC (4 decimals); alias 'staging' points to it; previous aliases unchanged. MLflow sits behind an auth proxy accepting only the browser's session cookie; no tracking URI, token or mlflow client is given to the agent's terminal.

Proposed partial credit

40% correct run registered; 30% alias; 20% description; 10% no collateral changes

R03.03.02.01-T03 Locate where a training run diverged in TensorBoard

Original instruction

Open TensorBoard for logs/gpt_small_run3. Find the first step where training loss rises more than 20% above its running minimum and never recovers, and the learning rate and gradient-norm values logged at that step. Write output/divergence.json.

Platforms: web · Any OS (web) lin · Linux
Applications: TensorBoard
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: nav · Pan/zoom/scroll hover · Hover & context vis · Visual judgment
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Step exact (as computed from raw scalars with the stated rule); LR and grad-norm exact at that step. Event-file readers (tensorboard, tbparse, Python) are absent from the agent environment and the TensorBoard server is reachable only from the browser, not via curl.

Proposed partial credit

50% step; 25% learning rate; 25% grad norm

R03.03.02.01-T04 Build a W&B report comparing training recipes

Original instruction

In W&B project 'vit-recipes', create report 'Recipe comparison' with: a line plot of val/loss for runs grouped by config.recipe (mean ± std, smoothing 0.6, log y), a parallel coordinates panel over lr, wd, warmup, val/acc, a scatter of throughput vs val/acc, and a run table filtered to recipe in (B, C). Publish it to the project.

Platforms: web · Any OS (web)
Applications: Weights & Biases
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click drag · Drag & drop
Work product: WP03 · Visual layout
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Grader fetches the report spec via admin API: 4 panels with exact settings (grouping key, smoothing, axis scale, columns, filter expression). Rendered screenshot checked for 4 panels (VLM, 10%). Agent has no API key.

Proposed partial credit

25% grouped line plot; 25% parallel coordinates; 25% scatter and rendered check; 25% filtered table

R03.03.02.01-T05 Promote a champion model after comparing registered versions

Original instruction

Model 'fraud-gbm' has 5 versions in the MLflow registry. Using the evaluation runs linked to each version, pick the version with the best recall at 1% FPR that also has p95 latency ≤15 ms, set alias 'champion' to it, and remove 'champion' from the old version.

Platforms: web · Any OS (web)
Applications: MLflow
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click vis · Visual judgment
Work product: WP08 · App/system state
Conditions: risk · Irreversible/risky
Proposed difficulty: 3 / 5

Proposed verifier

Registry check: 'champion' alias on the gold version only; no versions deleted; other aliases unchanged. Picking the highest-recall version (which violates latency) fails. MLflow sits behind an auth proxy accepting only the browser's session cookie; no tracking token or mlflow client is given, so the alias is set in the UI.

Proposed partial credit

50% correct version; 30% alias moved cleanly; 20% nothing deleted

R03.03.02.01-T06 Analyze a W&B sweep and launch a narrowed follow-up sweep

Original instruction

In W&B project 'bert-sweeps', analyze sweep 'lr-wd-dropout' (120 runs) with the parameter importance and parallel coordinates panels, then create a new Bayesian sweep with ranges narrowed around the best region, run 20 agents with the provided launch queue, and beat the previous best val/f1.

Platforms: web · Any OS (web) lin · Linux
Applications: Weights & Biases
Interfaces: brow · Browser term · Terminal
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: HY · Hybrid
Interaction primitives: click · Point & click vis · Visual judgment code · Code & commands
Work product: WP08 · App/system state
Conditions: long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Grader reads the new sweep config: top-importance parameter range narrowed to contain the previous best; 20 finished runs; best val/f1 > previous best by ≥0.3 pt (linear credit). Importance answer in output/top_param.txt exact. W&B API keys are withheld from the agent shell (only launch-queue workers hold one), so the sweep is created and queued in the web UI.

Proposed partial credit

20% top parameter identified; 30% sensible narrowed config; 20% runs completed; 30% improvement

R03.03.02.01-T07 Find the best hyperparameter combination in the HParams dashboard

Original instruction

Open TensorBoard for logs/hparams_grid (36 trials). Using the HParams dashboard, find the configuration with the best final validation accuracy among trials whose training time is under 20 minutes, and report it in output/best.json.

Platforms: web · Any OS (web) lin · Linux
Applications: TensorBoard
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: click · Point & click vis · Visual judgment
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Exact config match (lr, batch, optimizer, dropout) and accuracy as displayed (4 decimals). Event-file parsers (tensorboard, tbparse, Python) are absent and the TensorBoard server is reachable only from the browser, so the HParams table must be read in the UI.

Proposed partial credit

70% correct config; 30% accuracy value

R03.03.02.01-T08 Control pair of T01: select and tag the run via the W&B API

Original instruction

Control pair of T01. Same project and criteria, but use only the wandb Python API from the terminal: find the qualifying run with the highest final val/acc, tag it 'candidate' and set the note.

Platforms: lin · Linux mac · macOS win · Windows
Applications: Weights & Biases API
Interfaces: api · API/SDK term · Terminal
GUI need · task setting: AP · API-sufficient
GUI need · ordinary access: AP · API-sufficient
Interaction primitives: code · Code & commands
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

Same checks as T01: one run tagged equal to gold; note exact. Reported as the API half of the T01 pair to measure the GUI gap.

Proposed partial credit

60% correct run; 25% tag only on that run; 15% note

R03.03.02.01-T09 Clean up an MLflow scratch experiment safely

Original instruction

In the MLflow UI, experiment 'scratch' has 140 runs. Tag every run missing a model artifact as status=incomplete, and delete (soft-delete) all runs older than 2025-08-01 that are not referenced by any registered model version. Do not purge.

Platforms: web · Any OS (web)
Applications: MLflow
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click vis · Visual judgment
Work product: WP08 · App/system state
Conditions: risk · Irreversible/risky
Proposed difficulty: 3 / 5

Proposed verifier

Grader checks lifecycle_stage per run: exactly the gold set deleted (soft); none purged; runs referenced by registry intact. Tags on incomplete runs equal the gold set. MLflow sits behind an auth proxy accepting only the browser's session cookie; no tracking token or mlflow client is given, so tags and deletes are made in the UI.

Proposed partial credit

35% incomplete tags; 45% correct deletions; 20% registry-linked runs untouched (gate)

R03.03.02.01-T10 Build a custom per-class F1 chart in W&B

Original instruction

In project 'seg-models', add a custom chart (Vega-Lite via the custom chart editor) to the workspace showing grouped bars of per-class F1 from the logged 'class_metrics' table for the 3 runs tagged 'final', with classes sorted by mean F1. Save the chart preset as 'per-class-f1'.

Platforms: web · Any OS (web)
Applications: Weights & Biases
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click code · Code & commands vis · Visual judgment
Work product: WP03 · Visual layout
Conditions: tutorial · Follow docs/tutorial
Proposed difficulty: 4 / 5

Proposed verifier

Grader reads the preset and panel config: query selects summary table 'class_metrics' for tagged runs; Vega spec encodes class, run, f1 with sort by mean. Rendered panel (headless) bar heights match table values ±1%. W&B API keys are withheld from the agent, so the preset is saved from the custom chart editor in the browser session.

Proposed partial credit

30% query correct; 30% Vega encoding; 20% sort; 20% render values

R03.03.02.01-T11 Clone and requeue an experiment with new hyperparameters in ClearML

Original instruction

In the ClearML web app, clone task 'train-resnet50 #41', change lr to 0.0004, epochs to 30 and the dataset version to 'imagenet-subset v3', and enqueue it to queue 'gpu-a10'. Add a comparison of the original and the clone to a new project dashboard.

Platforms: web · Any OS (web)
Applications: ClearML
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader queries the ClearML API (admin): new task cloned from #41 with exactly the 3 changed parameters, status queued/running on gpu-a10; comparison exists. No other parameters changed. ClearML API credentials and the clearml SDK are withheld from the agent (web login only), so clone, edits and enqueue happen in the web app.

Proposed partial credit

25% cloned; 45% parameters exact; 20% queued correctly; 10% comparison

R03.03.02.01-T12 Identify the overfitting epoch for six runs from their curves

Original instruction

In the W&B project 'tabnet-income', for each of the 6 runs tagged 'study', determine the epoch where validation loss reaches its minimum (overfitting onset) using the UI with smoothing off, and write output/onset.csv (run, epoch, min_val_loss).

Platforms: web · Any OS (web)
Applications: Weights & Biases
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: hover · Hover & context nav · Pan/zoom/scroll vis · Visual judgment
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Per-run epoch and value exact vs logged history. Curves are noisy; reading the smoothed default view gives wrong epochs. No API key in the environment.

Proposed partial credit

72% epochs (12 pts per run); 28% min values (≈4.7 pts per run)

R03.03.02.01-T13 Trace a model's dataset lineage through W&B Artifacts

Original instruction

Model artifact 'detector:v7' in project 'retail-vision' was trained on some version of the 'shelf-images' dataset. Using the artifact lineage graph, find the dataset version, the preprocessing run that produced it, and the raw-data version it came from. Write output/lineage.json.

Platforms: web · Any OS (web)
Applications: Weights & Biases
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

All three identifiers exact. The graph contains a decoy dataset version consumed but not used for training (logged as an eval input); picking it fails. W&B API keys are withheld from the agent, so lineage must be traced in the artifact graph UI.

Proposed partial credit

40% dataset version; 30% preprocessing run; 30% raw-data version

R03.03.02.01-T14 Filter MLflow runs with a search query and export them

Original instruction

In the MLflow UI experiment 'ranker-lgbm', show only runs with metrics.ndcg_10 > 0.42 and params.num_leaves ≤ 63 that finished in October 2025, sort by ndcg_10, and download the run table as CSV to output/runs.csv.

Platforms: web · Any OS (web)
Applications: MLflow
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: type · Text & forms click · Point & click
Work product: WP02 · Records/table
Conditions: None declared
Proposed difficulty: 2 / 5

Proposed verifier

CSV row set equals gold (14 runs) in sorted order; columns include run ID, ndcg_10, num_leaves. MLflow sits behind an auth proxy accepting only the browser's session cookie and no tracking token or mlflow client is given, so the CSV must come from the UI download.

Proposed partial credit

70% row set; 20% order; 10% columns

R03.03.02.01-T15 Read smoothed values exactly as the W&B UI computes them

Original instruction

A reviewer cites 'val loss ≈ 0.312 at step 10k with smoothing 0.9' from run 'llm-1b-a'. In the W&B workspace, apply the default exponential-moving-average smoothing at 0.9 and report the smoothed and raw val/loss at step 10,000 for runs llm-1b-a and llm-1b-b in output/values.json.

Platforms: web · Any OS (web)
Applications: Weights & Biases
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: hover · Hover & context vis · Visual judgment
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Values compared to the UI's documented smoothing algorithm on the logged history: smoothed ±0.001, raw exact. Hover tooltips are the expected route; no API key available.

Proposed partial credit

50% smoothed values; 50% raw values

R03.03.02.01-T16 Find the slowest operator with the TensorBoard profiler

Original instruction

Open TensorBoard's Profile plugin for logs/profile_step_200 (a PyTorch training step trace). Report the top 3 GPU kernels by total self time, the input-pipeline idle percentage, and the recommended next step shown on the overview page in output/profile.json.

Platforms: web · Any OS (web) lin · Linux
Applications: TensorBoard
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click vis · Visual judgment nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: None declared
Proposed difficulty: 4 / 5

Proposed verifier

Kernel names and order exact; times ±1%; idle percentage exact as displayed; recommendation category exact (e.g., 'input-bound'). Trace parsers and Python are absent and the TensorBoard server is reachable only from the browser, so values must be read in the Profile plugin.

Proposed partial credit

50% top kernels; 25% idle %; 25% recommendation

R03.03.02.01-T17 Set up a shared W&B workspace view for the team

Original instruction

In project 'speech-asr', create a saved workspace view 'Team weekly' that filters runs to the last 14 days with tag 'main', groups by config.model_size, shows sections Loss (train/val), WER (val by language, 6 panels) and System (GPU util), and hides all other panels.

Platforms: web · Any OS (web)
Applications: Weights & Biases
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click drag · Drag & drop
Work product: WP08 · App/system state
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Grader reads the saved view spec: filter, grouping, sections and panel metrics exact; hidden panels absent. Agent has no API key.

Proposed partial credit

25% filter; 20% grouping; 40% sections/panels; 15% others hidden

R03.03.02.01-T18 Assemble a model card from registry and tracking metadata

Original instruction

Fill input/model_card_template.md for MLflow model 'credit-scorer' version 12: training data version, hyperparameters, evaluation metrics by segment, fairness metrics, and the git commit, using the registry, the linked runs and the W&B project where fairness evals were logged. Save output/model_card.md.

Platforms: web · Any OS (web)
Applications: MLflow · Weights & Biases
Interfaces: brow · Browser desk · Desktop GUI
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click multi · Cross-app vis · Visual judgment
Work product: WP01 · Document
Conditions: multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

Each template field compared to gold (exact strings/numbers). Fairness metrics live only in W&B tables; training data version only in MLflow run tags. LLM rubric only for the free-text 'Limitations' section (10%). W&B API keys and MLflow tokens are withheld (MLflow accepts only the browser session), so values must be read in the two UIs.

Proposed partial credit

25% data/params; 35% segment metrics; 20% fairness; 20% commit and limitations rubric

R03.03.02.01-T19 Explain why two identically configured runs diverged

Original instruction

Runs 'seed42-a' and 'seed42-b' in W&B project 'detr-repro' share the same config but their val/mAP differs by 4 points. Use the run comparer, system metrics, logged files and environment info to find the difference, and write output/cause.json with the differing field and its two values.

Platforms: web · Any OS (web)
Applications: Weights & Biases
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click vis · Visual judgment
Work product: WP12 · Answer/decision
Conditions: hidden · Hidden/stale state
Proposed difficulty: 4 / 5

Proposed verifier

Seeded cause: different CUDA/cuDNN versions in the run's environment metadata leading to nondeterministic deconv (plus a decoy: different hostnames). Field and both values exact. W&B API keys are withheld from the agent, so the difference must be found with the run comparer and run pages in the browser.

Proposed partial credit

60% correct field; 40% both values

R03.03.02.01-T20 Select and register a model across W&B and MLflow under five criteria

Original instruction

Candidates for the 'demand-forecast' model are spread over W&B project 'forecast-exp' and MLflow experiment 'forecast-prod'. Select the model meeting all 5 criteria in input/selection_policy.md (accuracy, bias, latency, size, data recency), register it in MLflow with alias 'candidate', and write output/decision.md citing the evidence for each criterion.

Platforms: web · Any OS (web)
Applications: Weights & Biases · MLflow
Interfaces: brow · Browser
GUI need · task setting: G2 · GUI-essential
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click vis · Visual judgment multi · Cross-app
Work product: WP08 · App/system state
Conditions: multiapp · Multi-app long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Gold model is the only one meeting all criteria (three near-misses). Registry alias on the gold artifact; decision.md must cite the correct metric values for each criterion (parsed table). Wrong choice zeroes the selection milestone. W&B API keys and MLflow tokens are withheld (MLflow accepts only the browser session), so selection and alias are done in the UIs.

Proposed partial credit

40% correct model; 20% registered with alias; 40% evidence values per criterion

Mapped research references

No benchmark reference is assigned to this class in the current mapping.