R03.03.02.01 · 任务小类

实验追踪与模型注册

Experiment tracking & model registry

Compare runs, select and register models, and read curves in tracking UIs.

20 条本项目编写的提案 · 分类 v1.1 · 新增基准运行 0 次。下方验证方式与部分得分规则是设计，不是执行结果。

典型应用: Weights & Biases · MLflow · TensorBoard
小类 GUI 需求标签: G2 · GUI 必要

任务提案 · 20

展开查看完整原文与建议检查方式；两种界面均保留提案英文原文。

R03.03.02.01-T01 Pick and tag the best qualifying run in a W&B project

原始指令

In the W&B sandbox project 'resnet-ablations' (48 runs), find the run with the highest final val/acc among runs with batch_size 64 whose final train/acc − val/acc gap is under 2 points. Tag it 'candidate' and add the note 'selected for eval' to it.

平台: web · 任意系统（网页）
应用: Weights & Biases
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 vis · 视觉判断
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader reads run tags and notes with an admin key: exactly one run tagged 'candidate', equal to gold; note text exact. Agent session has no API key and the wandb SDK is not installed; tag edits must come from the web UI (audit log user agent).

建议部分得分

60% correct run; 25% tag only on that run; 15% note

R03.03.02.01-T02 Register the best MLflow run as a new model version in Staging

原始指令

In the local MLflow UI, compare runs in experiment 'churn-xgb' and register the model artifact of the run with the highest val_auc as model 'churn-model' (new version) with description 'val_auc=<value>', then set alias 'staging' on it.

平台: web · 任意系统（网页）
应用: MLflow
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 2 / 5

建议验证方式

Grader queries the registry: new version source run_id equals the best run; description contains the exact AUC (4 decimals); alias 'staging' points to it; previous aliases unchanged. MLflow sits behind an auth proxy accepting only the browser's session cookie; no tracking URI, token or mlflow client is given to the agent's terminal.

建议部分得分

40% correct run registered; 30% alias; 20% description; 10% no collateral changes

R03.03.02.01-T03 Locate where a training run diverged in TensorBoard

原始指令

Open TensorBoard for logs/gpt_small_run3. Find the first step where training loss rises more than 20% above its running minimum and never recovers, and the learning rate and gradient-norm values logged at that step. Write output/divergence.json.

平台: web · 任意系统（网页） lin · Linux
应用: TensorBoard
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: nav · 平移 / 缩放 / 滚动 hover · 悬停与上下文 vis · 视觉判断
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Step exact (as computed from raw scalars with the stated rule); LR and grad-norm exact at that step. Event-file readers (tensorboard, tbparse, Python) are absent from the agent environment and the TensorBoard server is reachable only from the browser, not via curl.

建议部分得分

50% step; 25% learning rate; 25% grad norm

R03.03.02.01-T04 Build a W&B report comparing training recipes

原始指令

In W&B project 'vit-recipes', create report 'Recipe comparison' with: a line plot of val/loss for runs grouped by config.recipe (mean ± std, smoothing 0.6, log y), a parallel coordinates panel over lr, wd, warmup, val/acc, a scatter of throughput vs val/acc, and a run table filtered to recipe in (B, C). Publish it to the project.

平台: web · 任意系统（网页）
应用: Weights & Biases
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 drag · 拖放
工作产物: WP03 · 视觉排版
条件: 未声明
提案难度: 4 / 5

建议验证方式

Grader fetches the report spec via admin API: 4 panels with exact settings (grouping key, smoothing, axis scale, columns, filter expression). Rendered screenshot checked for 4 panels (VLM, 10%). Agent has no API key.

建议部分得分

25% grouped line plot; 25% parallel coordinates; 25% scatter and rendered check; 25% filtered table

R03.03.02.01-T05 Promote a champion model after comparing registered versions

原始指令

Model 'fraud-gbm' has 5 versions in the MLflow registry. Using the evaluation runs linked to each version, pick the version with the best recall at 1% FPR that also has p95 latency ≤15 ms, set alias 'champion' to it, and remove 'champion' from the old version.

平台: web · 任意系统（网页）
应用: MLflow
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 vis · 视觉判断
工作产物: WP08 · 应用 / 系统状态
条件: risk · 高风险 / 不可逆
提案难度: 3 / 5

建议验证方式

Registry check: 'champion' alias on the gold version only; no versions deleted; other aliases unchanged. Picking the highest-recall version (which violates latency) fails. MLflow sits behind an auth proxy accepting only the browser's session cookie; no tracking token or mlflow client is given, so the alias is set in the UI.

建议部分得分

50% correct version; 30% alias moved cleanly; 20% nothing deleted

R03.03.02.01-T06 Analyze a W&B sweep and launch a narrowed follow-up sweep

原始指令

In W&B project 'bert-sweeps', analyze sweep 'lr-wd-dropout' (120 runs) with the parameter importance and parallel coordinates panels, then create a new Bayesian sweep with ranges narrowed around the best region, run 20 agents with the provided launch queue, and beat the previous best val/f1.

平台: web · 任意系统（网页） lin · Linux
应用: Weights & Biases
交互表面: brow · 浏览器 term · 终端
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: HY · 混合
交互原语: click · 指向与点击 vis · 视觉判断 code · 代码与命令
工作产物: WP08 · 应用 / 系统状态
条件: long · 长流程
提案难度: 5 / 5

建议验证方式

Grader reads the new sweep config: top-importance parameter range narrowed to contain the previous best; 20 finished runs; best val/f1 > previous best by ≥0.3 pt (linear credit). Importance answer in output/top_param.txt exact. W&B API keys are withheld from the agent shell (only launch-queue workers hold one), so the sweep is created and queued in the web UI.

建议部分得分

20% top parameter identified; 30% sensible narrowed config; 20% runs completed; 30% improvement

R03.03.02.01-T07 Find the best hyperparameter combination in the HParams dashboard

原始指令

Open TensorBoard for logs/hparams_grid (36 trials). Using the HParams dashboard, find the configuration with the best final validation accuracy among trials whose training time is under 20 minutes, and report it in output/best.json.

平台: web · 任意系统（网页） lin · Linux
应用: TensorBoard
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: click · 指向与点击 vis · 视觉判断
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Exact config match (lr, batch, optimizer, dropout) and accuracy as displayed (4 decimals). Event-file parsers (tensorboard, tbparse, Python) are absent and the TensorBoard server is reachable only from the browser, so the HParams table must be read in the UI.

建议部分得分

70% correct config; 30% accuracy value

R03.03.02.01-T08 Control pair of T01: select and tag the run via the W&B API

原始指令

Control pair of T01. Same project and criteria, but use only the wandb Python API from the terminal: find the qualifying run with the highest final val/acc, tag it 'candidate' and set the note.

平台: lin · Linux mac · macOS win · Windows
应用: Weights & Biases API
交互表面: api · API / SDK term · 终端
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: AP · API 足够
交互原语: code · 代码与命令
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 2 / 5

建议验证方式

Same checks as T01: one run tagged equal to gold; note exact. Reported as the API half of the T01 pair to measure the GUI gap.

建议部分得分

60% correct run; 25% tag only on that run; 15% note

R03.03.02.01-T09 Clean up an MLflow scratch experiment safely

原始指令

In the MLflow UI, experiment 'scratch' has 140 runs. Tag every run missing a model artifact as status=incomplete, and delete (soft-delete) all runs older than 2025-08-01 that are not referenced by any registered model version. Do not purge.

平台: web · 任意系统（网页）
应用: MLflow
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 vis · 视觉判断
工作产物: WP08 · 应用 / 系统状态
条件: risk · 高风险 / 不可逆
提案难度: 3 / 5

建议验证方式

Grader checks lifecycle_stage per run: exactly the gold set deleted (soft); none purged; runs referenced by registry intact. Tags on incomplete runs equal the gold set. MLflow sits behind an auth proxy accepting only the browser's session cookie; no tracking token or mlflow client is given, so tags and deletes are made in the UI.

建议部分得分

35% incomplete tags; 45% correct deletions; 20% registry-linked runs untouched (gate)

R03.03.02.01-T10 Build a custom per-class F1 chart in W&B

原始指令

In project 'seg-models', add a custom chart (Vega-Lite via the custom chart editor) to the workspace showing grouped bars of per-class F1 from the logged 'class_metrics' table for the 3 runs tagged 'final', with classes sorted by mean F1. Save the chart preset as 'per-class-f1'.

平台: web · 任意系统（网页）
应用: Weights & Biases
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 code · 代码与命令 vis · 视觉判断
工作产物: WP03 · 视觉排版
条件: tutorial · 阅读文档 / 教程
提案难度: 4 / 5

建议验证方式

Grader reads the preset and panel config: query selects summary table 'class_metrics' for tagged runs; Vega spec encodes class, run, f1 with sort by mean. Rendered panel (headless) bar heights match table values ±1%. W&B API keys are withheld from the agent, so the preset is saved from the custom chart editor in the browser session.

建议部分得分

30% query correct; 30% Vega encoding; 20% sort; 20% render values

R03.03.02.01-T11 Clone and requeue an experiment with new hyperparameters in ClearML

原始指令

In the ClearML web app, clone task 'train-resnet50 #41', change lr to 0.0004, epochs to 30 and the dataset version to 'imagenet-subset v3', and enqueue it to queue 'gpu-a10'. Add a comparison of the original and the clone to a new project dashboard.

平台: web · 任意系统（网页）
应用: ClearML
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader queries the ClearML API (admin): new task cloned from #41 with exactly the 3 changed parameters, status queued/running on gpu-a10; comparison exists. No other parameters changed. ClearML API credentials and the clearml SDK are withheld from the agent (web login only), so clone, edits and enqueue happen in the web app.

建议部分得分

25% cloned; 45% parameters exact; 20% queued correctly; 10% comparison

R03.03.02.01-T12 Identify the overfitting epoch for six runs from their curves

原始指令

In the W&B project 'tabnet-income', for each of the 6 runs tagged 'study', determine the epoch where validation loss reaches its minimum (overfitting onset) using the UI with smoothing off, and write output/onset.csv (run, epoch, min_val_loss).

平台: web · 任意系统（网页）
应用: Weights & Biases
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: hover · 悬停与上下文 nav · 平移 / 缩放 / 滚动 vis · 视觉判断
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 4 / 5

建议验证方式

Per-run epoch and value exact vs logged history. Curves are noisy; reading the smoothed default view gives wrong epochs. No API key in the environment.

建议部分得分

72% epochs (12 pts per run); 28% min values (≈4.7 pts per run)

R03.03.02.01-T13 Trace a model's dataset lineage through W&B Artifacts

原始指令

Model artifact 'detector:v7' in project 'retail-vision' was trained on some version of the 'shelf-images' dataset. Using the artifact lineage graph, find the dataset version, the preprocessing run that produced it, and the raw-data version it came from. Write output/lineage.json.

平台: web · 任意系统（网页）
应用: Weights & Biases
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 nav · 平移 / 缩放 / 滚动
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

All three identifiers exact. The graph contains a decoy dataset version consumed but not used for training (logged as an eval input); picking it fails. W&B API keys are withheld from the agent, so lineage must be traced in the artifact graph UI.

建议部分得分

40% dataset version; 30% preprocessing run; 30% raw-data version

R03.03.02.01-T14 Filter MLflow runs with a search query and export them

原始指令

In the MLflow UI experiment 'ranker-lgbm', show only runs with metrics.ndcg_10 > 0.42 and params.num_leaves ≤ 63 that finished in October 2025, sort by ndcg_10, and download the run table as CSV to output/runs.csv.

平台: web · 任意系统（网页）
应用: MLflow
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: type · 文字与表单 click · 指向与点击
工作产物: WP02 · 记录 / 表格
条件: 未声明
提案难度: 2 / 5

建议验证方式

CSV row set equals gold (14 runs) in sorted order; columns include run ID, ndcg_10, num_leaves. MLflow sits behind an auth proxy accepting only the browser's session cookie and no tracking token or mlflow client is given, so the CSV must come from the UI download.

建议部分得分

70% row set; 20% order; 10% columns

R03.03.02.01-T15 Read smoothed values exactly as the W&B UI computes them

原始指令

A reviewer cites 'val loss ≈ 0.312 at step 10k with smoothing 0.9' from run 'llm-1b-a'. In the W&B workspace, apply the default exponential-moving-average smoothing at 0.9 and report the smoothed and raw val/loss at step 10,000 for runs llm-1b-a and llm-1b-b in output/values.json.

平台: web · 任意系统（网页）
应用: Weights & Biases
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: hover · 悬停与上下文 vis · 视觉判断
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Values compared to the UI's documented smoothing algorithm on the logged history: smoothed ±0.001, raw exact. Hover tooltips are the expected route; no API key available.

建议部分得分

50% smoothed values; 50% raw values

R03.03.02.01-T16 Find the slowest operator with the TensorBoard profiler

原始指令

Open TensorBoard's Profile plugin for logs/profile_step_200 (a PyTorch training step trace). Report the top 3 GPU kernels by total self time, the input-pipeline idle percentage, and the recommended next step shown on the overview page in output/profile.json.

平台: web · 任意系统（网页） lin · Linux
应用: TensorBoard
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 vis · 视觉判断 nav · 平移 / 缩放 / 滚动
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 4 / 5

建议验证方式

Kernel names and order exact; times ±1%; idle percentage exact as displayed; recommendation category exact (e.g., 'input-bound'). Trace parsers and Python are absent and the TensorBoard server is reachable only from the browser, so values must be read in the Profile plugin.

建议部分得分

50% top kernels; 25% idle %; 25% recommendation

R03.03.02.01-T17 Set up a shared W&B workspace view for the team

原始指令

In project 'speech-asr', create a saved workspace view 'Team weekly' that filters runs to the last 14 days with tag 'main', groups by config.model_size, shows sections Loss (train/val), WER (val by language, 6 panels) and System (GPU util), and hides all other panels.

平台: web · 任意系统（网页）
应用: Weights & Biases
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 drag · 拖放
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader reads the saved view spec: filter, grouping, sections and panel metrics exact; hidden panels absent. Agent has no API key.

建议部分得分

25% filter; 20% grouping; 40% sections/panels; 15% others hidden

R03.03.02.01-T18 Assemble a model card from registry and tracking metadata

原始指令

Fill input/model_card_template.md for MLflow model 'credit-scorer' version 12: training data version, hyperparameters, evaluation metrics by segment, fairness metrics, and the git commit, using the registry, the linked runs and the W&B project where fairness evals were logged. Save output/model_card.md.

平台: web · 任意系统（网页）
应用: MLflow · Weights & Biases
交互表面: brow · 浏览器 desk · 桌面 GUI
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 multi · 跨应用 vis · 视觉判断
工作产物: WP01 · 文档
条件: multiapp · 多应用
提案难度: 4 / 5

建议验证方式

Each template field compared to gold (exact strings/numbers). Fairness metrics live only in W&B tables; training data version only in MLflow run tags. LLM rubric only for the free-text 'Limitations' section (10%). W&B API keys and MLflow tokens are withheld (MLflow accepts only the browser session), so values must be read in the two UIs.

建议部分得分

25% data/params; 35% segment metrics; 20% fairness; 20% commit and limitations rubric

R03.03.02.01-T19 Explain why two identically configured runs diverged

原始指令

Runs 'seed42-a' and 'seed42-b' in W&B project 'detr-repro' share the same config but their val/mAP differs by 4 points. Use the run comparer, system metrics, logged files and environment info to find the difference, and write output/cause.json with the differing field and its two values.

平台: web · 任意系统（网页）
应用: Weights & Biases
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 vis · 视觉判断
工作产物: WP12 · 答案 / 决策
条件: hidden · 隐藏 / 过期状态
提案难度: 4 / 5

建议验证方式

Seeded cause: different CUDA/cuDNN versions in the run's environment metadata leading to nondeterministic deconv (plus a decoy: different hostnames). Field and both values exact. W&B API keys are withheld from the agent, so the difference must be found with the run comparer and run pages in the browser.

建议部分得分

60% correct field; 40% both values

R03.03.02.01-T20 Select and register a model across W&B and MLflow under five criteria

原始指令

Candidates for the 'demand-forecast' model are spread over W&B project 'forecast-exp' and MLflow experiment 'forecast-prod'. Select the model meeting all 5 criteria in input/selection_policy.md (accuracy, bias, latency, size, data recency), register it in MLflow with alias 'candidate', and write output/decision.md citing the evidence for each criterion.

平台: web · 任意系统（网页）
应用: Weights & Biases · MLflow
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 vis · 视觉判断 multi · 跨应用
工作产物: WP08 · 应用 / 系统状态
条件: multiapp · 多应用 long · 长流程
提案难度: 5 / 5

建议验证方式

Gold model is the only one meeting all criteria (three near-misses). Registry alias on the gold artifact; decision.md must cite the correct metric values for each criterion (parsed table). Wrong choice zeroes the selection milestone. W&B API keys and MLflow tokens are withheld (MLflow accepts only the browser session), so selection and alias are done in the UIs.

建议部分得分

40% correct model; 20% registered with alias; 40% evidence values per criterion

已有研究映射

当前映射未给此小类指定基准参考。