Annotation project setup & QA review
Configure labeling projects, label schemas and review queues, and adjudicate disagreements.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- CVAT · Label Studio · Labelbox
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R03.08.01.04-T01 CVAT shelf-audit project with attributes and job split
Original instruction
In self-hosted CVAT 2.x create project 'Retail-Shelf' with labels: product (rectangle; attributes brand select [Acme, Zeta, Other], facing checkbox default false) and price_tag (polygon). Create task 'store-07' from the 120 images in share/store07/, segment size 20, and assign jobs 1-3 to annotator ana and 4-6 to annotator ben.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader reads CVAT DB/API as admin: project labels and attribute specs exact (types, values, defaults); task size 120, 6 jobs of 20; assignees per job. The agent has only a browser session; requests carrying Authorization: Token or from cvat-cli are detected in the server access log -> 0.
Proposed partial credit
40% labels and attributes exact; 25% task with correct segment size; 35% job assignments
R03.08.01.04-T02 Label Studio NER project with relations and overlap
Original instruction
In Label Studio 1.13 create project 'Contracts-NER': labeling interface with span labels PARTY, DATE, AMOUNT, JURISDICTION (colors from fixtures/colors.json) and a relation 'governs' between JURISDICTION and PARTY. Import fixtures/contracts_200.json, and set 2 annotators per task with 'Show only unannotated' distribution.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader GET project as admin: parsed label_config has the 4 labels with exact colors and a Relations block with 'governs'; 200 tasks; maximum_annotations=2; sampling setting. Legacy API tokens are disabled for the agent's user; token-authenticated requests are detected in the nginx log -> 0.
Proposed partial credit
40% label config exact; 20% relation defined; 20% tasks imported; 20% overlap and distribution
R03.08.01.04-T03 Review a CVAT job and pin issues on wrong boxes
Original instruction
Job 14 of task 'traffic-cam' is in validation. Following fixtures/review_guide.pdf, raise an issue on each of the 6 incorrect boxes with a comment naming the error type (wrong class / loose box / missing occluded flag), placing the issue on the object, then set the job to 'rejected' and leave the 40 correct boxes untouched.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader: job issues = 6, each issue position lies inside a seeded wrong box (point-in-box) with comment type matching gold; no issues on correct boxes; annotations unchanged (hash); job state rejected. Browser-session-only access; API token use detected in the access log -> 0.
Proposed partial credit
50% six issues located on the right boxes; 25% error types correct; 10% no annotation edits; 15% job rejected
R03.08.01.04-T04 Adjudicate low-agreement Label Studio tasks
Original instruction
Project 'Support-Intent' has 40 tasks with 3 annotations each. For every task where the annotators disagree, choose the annotation that follows fixtures/guidelines.md (or correct one), mark it as ground truth, and leave unanimous tasks alone. Then export the ground-truth set as JSON-MIN to ~/out/gt.json.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader: tasks with disagreement (17) each have exactly one ground_truth annotation whose label equals gold adjudication (accuracy >= 0.9); unanimous tasks unchanged; gt.json matches the DB. Token-authenticated API calls detected in nginx log -> 0.
Proposed partial credit
20% only disputed tasks touched; 60% adjudication accuracy; 20% export matches
R03.08.01.04-T05 Labelbox ontology with nested classifications and review steps
Original instruction
In the Labelbox sandbox org, create ontology 'Defects-v2': bounding box 'scratch' with nested radio 'severity' (minor, major), polygon 'dent', global checklist 'lighting' (glare, shadow, ok). Create project 'line-3', attach the ontology, add dataset 'line3_oct' as a batch with priority 2, and configure workflow Initial review -> Rework with 1 reviewer.
- Platforms
web· Any OS (web)- Applications
- Labelbox
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
Grader via Labelbox SDK with an admin key (not given to agent): ontology normalized JSON equals gold; project has the ontology, one batch with priority 2 and all 300 data rows; workflow tasks exact. Org audit shows changes by the agent's user via web UI; API-key activity by that user detected -> 0.
Proposed partial credit
40% ontology exact incl. nesting; 25% project and batch; 35% workflow steps
R03.08.01.04-T06 Ground-truth job and quality report in CVAT
Original instruction
For CVAT task 'parking-lots' (500 frames) create a ground-truth job with 50 random frames (seed 7), annotate it from fixtures/gt_boxes.png overlays, set quality settings IoU threshold 0.5 and low-overlap 0.8, compute the quality report and write the task's overall accuracy to ~/out/quality.txt.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickdrag· Drag & dropgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Grader: GT job with 50 frames; its boxes match the gold boxes (mean IoU >= 0.85); quality settings exact; a quality report exists and quality.txt equals its accuracy to 3 dp. Browser-session-only; token API calls detected -> 0.
Proposed partial credit
20% GT job created; 40% GT boxes accurate; 15% quality settings; 25% report and value
R03.08.01.04-T07 Connect an ML backend for pre-annotations in Label Studio
Original instruction
Connect the ML backend at http://ml.test:9090 to project 'Product-Photos' in Label Studio, enable interactive pre-annotations, set the model version to 'yolo-v3-shelf', and retrieve predictions for all 50 tasks. Then accept predictions as annotations only for tasks with score >= 0.8.
- Platforms
web· Any OS (web)- Applications
- Label Studio · Label Studio ML backend
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & click- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader DB: ml_backend row with URL and is_interactive=true; model_version set; 50 predictions with that version; annotations exist exactly for the 31 tasks whose prediction score >= 0.8 and equal those predictions.
Proposed partial credit
30% backend connected and interactive; 30% predictions retrieved; 40% exactly the high-score tasks accepted
R03.08.01.04-T08 Fix a CVAT label schema without losing annotations
Original instruction
Project 'Street-v1' has a misspelled label 'persn' (1,240 boxes), and separate labels 'car' and 'auto' that should be one. Rename 'persn' to 'person', add attribute 'occluded' (checkbox, default false) to person, and move all 'auto' boxes to 'car' before deleting 'auto'.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Grader: label set equals the seed set with 'persn' renamed to 'person' and 'auto' removed; person box count 1,240 with same geometry hashes and occluded=false; car count = old car + old auto (2,311) with geometry preserved; no 'auto' label. Session-only access; token API or direct DB edits detected via access/audit log -> 0.
Proposed partial credit
25% rename kept all boxes; 20% attribute added; 45% auto boxes moved to car; 10% auto deleted last
R03.08.01.04-T09 Saved review tab for suspiciously fast annotations
Original instruction
In Label Studio project 'Reviews-Sentiment', create a Data Manager tab 'Fast negatives' filtering tasks where the annotation label is 'negative' and lead time < 5 s, with columns id, text, annotator, lead time, sorted by lead time ascending. Assign all its tasks to reviewer 'rita'.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader DB: view row titled 'Fast negatives' with filter items (annotation label, lead_time < 5) and ordering; selected columns; the 23 matching tasks have rita as reviewer assignee. Data Manager tabs are created via the UI session; token API calls detected -> 0.
Proposed partial credit
45% filter exact; 20% columns and sort; 35% reviewer assignment
R03.08.01.04-T10 Export CVAT subsets in two formats
Original instruction
From CVAT project 'Drones' export the 'validation' subset as COCO 1.0 with images to ~/out/val_coco.zip and the 'train' subset as YOLO 1.1 without images to ~/out/train_yolo.zip.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & click- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
val_coco.zip: instances JSON with 180 images and annotation count equal to DB, images included; train_yolo.zip: obj.names order matches project labels, 820 label files, no images. UI or API both acceptable for this task.
Proposed partial credit
50% COCO validation export correct; 50% YOLO train export correct
R03.08.01.04-T11 Consensus and benchmark setup in Labelbox
Original instruction
In Labelbox project 'line-3', enable consensus with 3 labelers on 20% of data rows, mark the 15 data rows listed in fixtures/benchmarks.csv as benchmarks using their reference labels, and after the grader's simulated labeling completes, export the agreement score per labeler to ~/out/agreement.csv.
- Platforms
web· Any OS (web)- Applications
- Labelbox
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & click- Work product
WP08· App/system state- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Grader SDK (admin key withheld from agent): project quality mode consensus, coverage 20%, 3 labelers; 15 benchmark rows equal fixture ids; agreement.csv matches the UI-computed scores within 0.01. API-key activity by the agent's user detected -> 0.
Proposed partial credit
35% consensus settings; 35% benchmarks set; 30% agreement exported
R03.08.01.04-T12 Fix ID switches in a CVAT video track job
Original instruction
Video task 'intersection-cam' job 3 (frames 100-300) has 3 ID switches where two pedestrians' tracks swap. Using the track tools (split, merge, keyframe edits), fix them so each person keeps one ID through the window, without changing boxes outside 100-300.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickdrag· Drag & dropav· Audio/video over timenav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Grader: IDF1 vs gold tracks on frames 100-300 >= 0.95 and 0 ID switches (MOT metrics); boxes outside the window unchanged (hash); per-frame mean IoU >= 0.85. Session-only; token API uploads detected in the access log -> 0.
Proposed partial credit
50% ID switches eliminated; 30% IoU kept; 20% outside window untouched
R03.08.01.04-T13 CVAT organization with members and roles
Original instruction
In CVAT create organization 'acme-labeling' (slug acme), invite users ana and ben as worker, carla as supervisor, and dan as maintainer, then move project 'Retail-Shelf' into the organization.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader DB: organization slug acme; memberships with exact roles (invitations accepted by grader bots); project organization_id = acme. Session-only; token-authenticated requests detected in the access log -> 0.
Proposed partial credit
20% org created; 50% four members with right roles; 30% project moved
R03.08.01.04-T14 S3-compatible storage sync and export target in Label Studio
Original instruction
In Label Studio project 'Aerial', add a source cloud storage pointing to MinIO bucket 'aerial-raw' (endpoint http://minio.test:9000, prefix 2026/10/, regex '.*\.jpg$', presigned URLs on) and a target storage to bucket 'aerial-labels'. Sync the source, then sync the target after the grader annotates 5 tasks.
- Platforms
web· Any OS (web)- Applications
- Label Studio · MinIO
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader: s3 import storage row with bucket, prefix, regex, presign=true; tasks imported = 240 (.jpg only, .png decoys excluded); export storage row; aerial-labels bucket contains 5 annotation JSON objects. Token API use by the agent's user detected -> 0.
Proposed partial credit
30% source storage settings; 30% correct tasks imported; 20% target storage; 20% annotations exported
R03.08.01.04-T15 Attach a guide and route finished jobs to validation
Original instruction
For CVAT project 'Retail-Shelf', write the annotation guide from fixtures/guide.md (including its 2 example images) into the project's guide editor, then move every job in state 'completed' to stage 'validation' and assign validator 'carla'.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsdrag· Drag & drop- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader: project annotation guide markdown contains all headings and both images as guide assets; the 4 completed jobs now stage=validation, state new, assignee carla; other jobs unchanged. Session-only; token API calls detected -> 0.
Proposed partial credit
40% guide with images; 45% completed jobs routed to validation; 15% other jobs untouched
R03.08.01.04-T16 Inter-annotator agreement report from a Label Studio export
Original instruction
Export project 'Toxicity-Pilot' (2 annotators on all 300 tasks) from Label Studio, compute Cohen's kappa overall and per label (toxic, insult, threat, none), write ~/out/kappa.json, and add a project-level 'Pilot QA' tab listing tasks where the two disagree.
- Platforms
web· Any OS (web)lin· Linux- Applications
- Label Studio · Python
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
click· Point & clickcode· Code & commandsmulti· Cross-app- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
kappa.json values within 0.005 of grader computation; Data Manager tab 'Pilot QA' filter returns exactly the 47 disagreeing tasks.
Proposed partial credit
30% overall kappa; 30% per-label kappa; 40% disagreement tab exact
R03.08.01.04-T17 Audio segmentation config with custom hotkeys
Original instruction
In Label Studio create project 'Call-Diarization' with an audio labeling interface: region labels AGENT (hotkey a), CUSTOMER (hotkey c), HOLD_MUSIC (hotkey h), plus a per-region text area 'transcript' and a choice 'language' (en, es). Import fixtures/calls/*.wav (12 files) and label call_01 fully per fixtures/call_01_gold.json timings.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickkey· Shortcutsav· Audio/video over timedrag· Drag & drop- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader: label config parsed (labels, hotkeys, perRegion textarea, choices); 12 tasks; call_01 regions match gold with boundary error <= 0.3 s and correct labels (segment F1 >= 0.9). Token API calls detected in nginx log -> 0.
Proposed partial credit
35% config incl. hotkeys; 15% files imported; 50% call_01 regions accurate
R03.08.01.04-T18 Build a 17-keypoint skeleton label in CVAT
Original instruction
In CVAT project 'Pose-Gym' add a skeleton label 'person' with the 17 COCO keypoints and 16 edges, laid out in the skeleton constructor to match fixtures/skeleton_template.png (positions within 5% of canvas), then annotate frame 0 of task 'gym-01' with one skeleton per visible person (3).
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryclick· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader: skeleton SVG in the label has 17 named points and 16 edges equal to COCO; normalized template positions within 0.05; frame 0 has 3 skeletons with OKS >= 0.75 vs gold. Session-only; token API uploads detected -> 0.
Proposed partial credit
30% points and names; 20% edges; 15% layout matches template; 35% three skeletons annotated
R03.08.01.04-T19 End-to-end annotation project from a guideline PDF
Original instruction
From fixtures/client_brief.pdf set up CVAT for a new client: project with the label schema the brief specifies (5 labels, 3 attributes), a task from share/client/ (600 images) with 30-image jobs, a 10% ground-truth job annotated by you, quality settings from the brief, assignment of 4 annotators round-robin, and a filled-in guide.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsdrag· Drag & dropvis· Visual judgment- Work product
WP08· App/system state- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader compares to the brief: label/attribute schema exact, 20 jobs, GT job 60 frames with mean IoU >= 0.8 vs gold, quality settings exact, round-robin assignments, guide headings present. Session-only; token API or cvat-cli usage detected in the access log -> 0.
Proposed partial credit
30% schema exact; 20% task and jobs; 30% GT job accurate; 20% quality, assignments and guide
R03.08.01.04-T20 Reject annotations with rubric reasons in a Label Studio review
Original instruction
As reviewer 'rita' in Label Studio project 'Receipts-OCR', review the 30 annotations in her queue against fixtures/rubric.md: accept compliant ones, reject the non-compliant ones with a comment citing the rubric rule number (e.g. 'R3: total missing').
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader DB: review states for 30 annotations vs gold (12 should be rejected) scored by accuracy; each rejection comment contains the correct rule id; rejected annotations returned to annotator queue. Token API calls by rita detected -> 0.
Proposed partial credit
60% accept/reject accuracy; 30% correct rule ids; 10% all 30 reviewed
Mapped research references
No benchmark reference is assigned to this class in the current mapping.