R03.08.01.04 · 任务小类

标注项目搭建与质检

Annotation project setup & QA review

Configure labeling projects, label schemas and review queues, and adjudicate disagreements.

20 条本项目编写的提案 · 分类 v1.1 · 新增基准运行 0 次。下方验证方式与部分得分规则是设计，不是执行结果。

典型应用: CVAT · Label Studio · Labelbox
小类 GUI 需求标签: G2 · GUI 必要

任务提案 · 20

展开查看完整原文与建议检查方式；两种界面均保留提案英文原文。

R03.08.01.04-T01 CVAT shelf-audit project with attributes and job split

原始指令

In self-hosted CVAT 2.x create project 'Retail-Shelf' with labels: product (rectangle; attributes brand select [Acme, Zeta, Other], facing checkbox default false) and price_tag (polygon). Create task 'store-07' from the 120 images in share/store07/, segment size 20, and assign jobs 1-3 to annotator ana and 4-6 to annotator ben.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader reads CVAT DB/API as admin: project labels and attribute specs exact (types, values, defaults); task size 120, 6 jobs of 20; assignees per job. The agent has only a browser session; requests carrying Authorization: Token or from cvat-cli are detected in the server access log -> 0.

建议部分得分

40% labels and attributes exact; 25% task with correct segment size; 35% job assignments

R03.08.01.04-T02 Label Studio NER project with relations and overlap

原始指令

In Label Studio 1.13 create project 'Contracts-NER': labeling interface with span labels PARTY, DATE, AMOUNT, JURISDICTION (colors from fixtures/colors.json) and a relation 'governs' between JURISDICTION and PARTY. Import fixtures/contracts_200.json, and set 2 annotators per task with 'Show only unannotated' distribution.

平台: web · 任意系统（网页）
应用: Label Studio
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader GET project as admin: parsed label_config has the 4 labels with exact colors and a Relations block with 'governs'; 200 tasks; maximum_annotations=2; sampling setting. Legacy API tokens are disabled for the agent's user; token-authenticated requests are detected in the nginx log -> 0.

建议部分得分

40% label config exact; 20% relation defined; 20% tasks imported; 20% overlap and distribution

R03.08.01.04-T03 Review a CVAT job and pin issues on wrong boxes

原始指令

Job 14 of task 'traffic-cam' is in validation. Following fixtures/review_guide.pdf, raise an issue on each of the 6 incorrect boxes with a comment naming the error type (wrong class / loose box / missing occluded flag), placing the issue on the object, then set the job to 'rejected' and leave the 40 correct boxes untouched.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单 vis · 视觉判断 nav · 平移 / 缩放 / 滚动
工作产物: WP10 · 标注
条件: 未声明
提案难度: 4 / 5

建议验证方式

Grader: job issues = 6, each issue position lies inside a seeded wrong box (point-in-box) with comment type matching gold; no issues on correct boxes; annotations unchanged (hash); job state rejected. Browser-session-only access; API token use detected in the access log -> 0.

建议部分得分

50% six issues located on the right boxes; 25% error types correct; 10% no annotation edits; 15% job rejected

R03.08.01.04-T04 Adjudicate low-agreement Label Studio tasks

原始指令

Project 'Support-Intent' has 40 tasks with 3 annotations each. For every task where the annotators disagree, choose the annotation that follows fixtures/guidelines.md (or correct one), mark it as ground truth, and leave unanimous tasks alone. Then export the ground-truth set as JSON-MIN to ~/out/gt.json.

平台: web · 任意系统（网页）
应用: Label Studio
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 vis · 视觉判断
工作产物: WP10 · 标注
条件: 未声明
提案难度: 4 / 5

建议验证方式

Grader: tasks with disagreement (17) each have exactly one ground_truth annotation whose label equals gold adjudication (accuracy >= 0.9); unanimous tasks unchanged; gt.json matches the DB. Token-authenticated API calls detected in nginx log -> 0.

建议部分得分

20% only disputed tasks touched; 60% adjudication accuracy; 20% export matches

R03.08.01.04-T05 Labelbox ontology with nested classifications and review steps

原始指令

In the Labelbox sandbox org, create ontology 'Defects-v2': bounding box 'scratch' with nested radio 'severity' (minor, major), polygon 'dent', global checklist 'lighting' (glare, shadow, ok). Create project 'line-3', attach the ontology, add dataset 'line3_oct' as a batch with priority 2, and configure workflow Initial review -> Rework with 1 reviewer.

平台: web · 任意系统（网页）
应用: Labelbox
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: licensed · 授权专业软件
提案难度: 4 / 5

建议验证方式

Grader via Labelbox SDK with an admin key (not given to agent): ontology normalized JSON equals gold; project has the ontology, one batch with priority 2 and all 300 data rows; workflow tasks exact. Org audit shows changes by the agent's user via web UI; API-key activity by that user detected -> 0.

建议部分得分

40% ontology exact incl. nesting; 25% project and batch; 35% workflow steps

R03.08.01.04-T06 Ground-truth job and quality report in CVAT

原始指令

For CVAT task 'parking-lots' (500 frames) create a ground-truth job with 50 random frames (seed 7), annotate it from fixtures/gt_boxes.png overlays, set quality settings IoU threshold 0.5 and low-overlap 0.8, compute the quality report and write the task's overall accuracy to ~/out/quality.txt.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 drag · 拖放 geom · 精确几何 vis · 视觉判断
工作产物: WP10 · 标注
条件: 未声明
提案难度: 5 / 5

建议验证方式

Grader: GT job with 50 frames; its boxes match the gold boxes (mean IoU >= 0.85); quality settings exact; a quality report exists and quality.txt equals its accuracy to 3 dp. Browser-session-only; token API calls detected -> 0.

建议部分得分

20% GT job created; 40% GT boxes accurate; 15% quality settings; 25% report and value

R03.08.01.04-T07 Connect an ML backend for pre-annotations in Label Studio

原始指令

Connect the ML backend at http://ml.test:9090 to project 'Product-Photos' in Label Studio, enable interactive pre-annotations, set the model version to 'yolo-v3-shelf', and retrieve predictions for all 50 tasks. Then accept predictions as annotations only for tasks with score >= 0.8.

平台: web · 任意系统（网页）
应用: Label Studio · Label Studio ML backend
交互表面: brow · 浏览器
GUI 需求 · 任务设定: HY · 混合
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击
工作产物: WP10 · 标注
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader DB: ml_backend row with URL and is_interactive=true; model_version set; 50 predictions with that version; annotations exist exactly for the 31 tasks whose prediction score >= 0.8 and equal those predictions.

建议部分得分

30% backend connected and interactive; 30% predictions retrieved; 40% exactly the high-score tasks accepted

R03.08.01.04-T08 Fix a CVAT label schema without losing annotations

原始指令

Project 'Street-v1' has a misspelled label 'persn' (1,240 boxes), and separate labels 'car' and 'auto' that should be one. Rename 'persn' to 'person', add attribute 'occluded' (checkbox, default false) to person, and move all 'auto' boxes to 'car' before deleting 'auto'.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP10 · 标注
条件: risk · 高风险 / 不可逆
提案难度: 4 / 5

建议验证方式

Grader: label set equals the seed set with 'persn' renamed to 'person' and 'auto' removed; person box count 1,240 with same geometry hashes and occluded=false; car count = old car + old auto (2,311) with geometry preserved; no 'auto' label. Session-only access; token API or direct DB edits detected via access/audit log -> 0.

建议部分得分

25% rename kept all boxes; 20% attribute added; 45% auto boxes moved to car; 10% auto deleted last

R03.08.01.04-T09 Saved review tab for suspiciously fast annotations

原始指令

In Label Studio project 'Reviews-Sentiment', create a Data Manager tab 'Fast negatives' filtering tasks where the annotation label is 'negative' and lead time < 5 s, with columns id, text, annotator, lead time, sorted by lead time ascending. Assign all its tasks to reviewer 'rita'.

平台: web · 任意系统（网页）
应用: Label Studio
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader DB: view row titled 'Fast negatives' with filter items (annotation label, lead_time < 5) and ordering; selected columns; the 23 matching tasks have rita as reviewer assignee. Data Manager tabs are created via the UI session; token API calls detected -> 0.

建议部分得分

45% filter exact; 20% columns and sort; 35% reviewer assignment

R03.08.01.04-T10 Export CVAT subsets in two formats

原始指令

From CVAT project 'Drones' export the 'validation' subset as COCO 1.0 with images to ~/out/val_coco.zip and the 'train' subset as YOLO 1.1 without images to ~/out/train_yolo.zip.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: HY · 混合
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击
工作产物: WP09 · 文件与文件夹
条件: 未声明
提案难度: 2 / 5

建议验证方式

val_coco.zip: instances JSON with 180 images and annotation count equal to DB, images included; train_yolo.zip: obj.names order matches project labels, 820 label files, no images. UI or API both acceptable for this task.

建议部分得分

50% COCO validation export correct; 50% YOLO train export correct

R03.08.01.04-T11 Consensus and benchmark setup in Labelbox

原始指令

In Labelbox project 'line-3', enable consensus with 3 labelers on 20% of data rows, mark the 15 data rows listed in fixtures/benchmarks.csv as benchmarks using their reference labels, and after the grader's simulated labeling completes, export the agreement score per labeler to ~/out/agreement.csv.

平台: web · 任意系统（网页）
应用: Labelbox
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击
工作产物: WP08 · 应用 / 系统状态
条件: licensed · 授权专业软件
提案难度: 3 / 5

建议验证方式

Grader SDK (admin key withheld from agent): project quality mode consensus, coverage 20%, 3 labelers; 15 benchmark rows equal fixture ids; agreement.csv matches the UI-computed scores within 0.01. API-key activity by the agent's user detected -> 0.

建议部分得分

35% consensus settings; 35% benchmarks set; 30% agreement exported

R03.08.01.04-T12 Fix ID switches in a CVAT video track job

原始指令

Video task 'intersection-cam' job 3 (frames 100-300) has 3 ID switches where two pedestrians' tracks swap. Using the track tools (split, merge, keyframe edits), fix them so each person keeps one ID through the window, without changing boxes outside 100-300.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 drag · 拖放 av · 音视频时序 nav · 平移 / 缩放 / 滚动
工作产物: WP10 · 标注
条件: 未声明
提案难度: 5 / 5

建议验证方式

Grader: IDF1 vs gold tracks on frames 100-300 >= 0.95 and 0 ID switches (MOT metrics); boxes outside the window unchanged (hash); per-frame mean IoU >= 0.85. Session-only; token API uploads detected in the access log -> 0.

建议部分得分

50% ID switches eliminated; 30% IoU kept; 20% outside window untouched

R03.08.01.04-T13 CVAT organization with members and roles

原始指令

In CVAT create organization 'acme-labeling' (slug acme), invite users ana and ben as worker, carla as supervisor, and dan as maintainer, then move project 'Retail-Shelf' into the organization.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 2 / 5

建议验证方式

Grader DB: organization slug acme; memberships with exact roles (invitations accepted by grader bots); project organization_id = acme. Session-only; token-authenticated requests detected in the access log -> 0.

建议部分得分

20% org created; 50% four members with right roles; 30% project moved

R03.08.01.04-T14 S3-compatible storage sync and export target in Label Studio

原始指令

In Label Studio project 'Aerial', add a source cloud storage pointing to MinIO bucket 'aerial-raw' (endpoint http://minio.test:9000, prefix 2026/10/, regex '.*\.jpg$', presigned URLs on) and a target storage to bucket 'aerial-labels'. Sync the source, then sync the target after the grader annotates 5 tasks.

平台: web · 任意系统（网页）
应用: Label Studio · MinIO
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader: s3 import storage row with bucket, prefix, regex, presign=true; tasks imported = 240 (.jpg only, .png decoys excluded); export storage row; aerial-labels bucket contains 5 annotation JSON objects. Token API use by the agent's user detected -> 0.

建议部分得分

30% source storage settings; 30% correct tasks imported; 20% target storage; 20% annotations exported

R03.08.01.04-T15 Attach a guide and route finished jobs to validation

原始指令

For CVAT project 'Retail-Shelf', write the annotation guide from fixtures/guide.md (including its 2 example images) into the project's guide editor, then move every job in state 'completed' to stage 'validation' and assign validator 'carla'.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单 drag · 拖放
工作产物: WP08 · 应用 / 系统状态
条件: 未声明
提案难度: 2 / 5

建议验证方式

Grader: project annotation guide markdown contains all headings and both images as guide assets; the 4 completed jobs now stage=validation, state new, assignee carla; other jobs unchanged. Session-only; token API calls detected -> 0.

建议部分得分

40% guide with images; 45% completed jobs routed to validation; 15% other jobs untouched

R03.08.01.04-T16 Inter-annotator agreement report from a Label Studio export

原始指令

Export project 'Toxicity-Pilot' (2 annotators on all 300 tasks) from Label Studio, compute Cohen's kappa overall and per label (toxic, insult, threat, none), write ~/out/kappa.json, and add a project-level 'Pilot QA' tab listing tasks where the two disagree.

平台: web · 任意系统（网页） lin · Linux
应用: Label Studio · Python
交互表面: brow · 浏览器 term · 终端
GUI 需求 · 任务设定: HY · 混合
GUI 需求 · 普通权限: HY · 混合
交互原语: click · 指向与点击 code · 代码与命令 multi · 跨应用
工作产物: WP02 · 记录 / 表格
条件: 未声明
提案难度: 3 / 5

建议验证方式

kappa.json values within 0.005 of grader computation; Data Manager tab 'Pilot QA' filter returns exactly the 47 disagreeing tasks.

建议部分得分

30% overall kappa; 30% per-label kappa; 40% disagreement tab exact

R03.08.01.04-T17 Audio segmentation config with custom hotkeys

原始指令

In Label Studio create project 'Call-Diarization' with an audio labeling interface: region labels AGENT (hotkey a), CUSTOMER (hotkey c), HOLD_MUSIC (hotkey h), plus a per-region text area 'transcript' and a choice 'language' (en, es). Import fixtures/calls/*.wav (12 files) and label call_01 fully per fixtures/call_01_gold.json timings.

平台: web · 任意系统（网页）
应用: Label Studio
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 key · 快捷键 av · 音视频时序 drag · 拖放
工作产物: WP10 · 标注
条件: 未声明
提案难度: 4 / 5

建议验证方式

Grader: label config parsed (labels, hotkeys, perRegion textarea, choices); 12 tasks; call_01 regions match gold with boundary error <= 0.3 s and correct labels (segment F1 >= 0.9). Token API calls detected in nginx log -> 0.

建议部分得分

35% config incl. hotkeys; 15% files imported; 50% call_01 regions accurate

R03.08.01.04-T18 Build a 17-keypoint skeleton label in CVAT

原始指令

In CVAT project 'Pose-Gym' add a skeleton label 'person' with the 17 COCO keypoints and 16 edges, laid out in the skeleton constructor to match fixtures/skeleton_template.png (positions within 5% of canvas), then annotate frame 0 of task 'gym-01' with one skeleton per visible person (3).

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: drag · 拖放 geom · 精确几何 click · 指向与点击 vis · 视觉判断
工作产物: WP10 · 标注
条件: 未声明
提案难度: 4 / 5

建议验证方式

Grader: skeleton SVG in the label has 17 named points and 16 edges equal to COCO; normalized template positions within 0.05; frame 0 has 3 skeletons with OKS >= 0.75 vs gold. Session-only; token API uploads detected -> 0.

建议部分得分

30% points and names; 20% edges; 15% layout matches template; 35% three skeletons annotated

R03.08.01.04-T19 End-to-end annotation project from a guideline PDF

原始指令

From fixtures/client_brief.pdf set up CVAT for a new client: project with the label schema the brief specifies (5 labels, 3 attributes), a task from share/client/ (600 images) with 30-image jobs, a 10% ground-truth job annotated by you, quality settings from the brief, assignment of 4 annotators round-robin, and a filled-in guide.

平台: web · 任意系统（网页）
应用: CVAT
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单 drag · 拖放 vis · 视觉判断
工作产物: WP08 · 应用 / 系统状态
条件: long · 长流程
提案难度: 5 / 5

建议验证方式

Grader compares to the brief: label/attribute schema exact, 20 jobs, GT job 60 frames with mean IoU >= 0.8 vs gold, quality settings exact, round-robin assignments, guide headings present. Session-only; token API or cvat-cli usage detected in the access log -> 0.

建议部分得分

30% schema exact; 20% task and jobs; 30% GT job accurate; 20% quality, assignments and guide

R03.08.01.04-T20 Reject annotations with rubric reasons in a Label Studio review

原始指令

As reviewer 'rita' in Label Studio project 'Receipts-OCR', review the 30 annotations in her queue against fixtures/rubric.md: accept compliant ones, reject the non-compliant ones with a comment citing the rubric rule number (e.g. 'R3: total missing').

平台: web · 任意系统（网页）
应用: Label Studio
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单 vis · 视觉判断
工作产物: WP10 · 标注
条件: 未声明
提案难度: 3 / 5

建议验证方式

Grader DB: review states for 30 annotations vs gold (12 should be rejected) scored by accuracy; each rejection comment contains the correct rule id; rejected annotations returned to annotator queue. Token API calls by rita detected -> 0.

标注项目搭建与质检

任务提案 · 20

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

已有研究映射

同工作流族的小类