Image boxes, polygons & masks
Label objects in images to an IoU threshold against reference annotations.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- CVAT · Label Studio · labelme
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R03.08.01.01-T01 Box every car in a traffic set (CVAT)
Original instruction
In CVAT, open the project 'traffic10' (10 street images) and draw a tight bounding box labeled 'car' around every car. Include partially visible cars at image edges. Export as COCO when done.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Exported COCO matched to hidden gold: mean IoU >= 0.7 and recall >= 0.9 for class car. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
40% recall of cars; 40% mean IoU >= 0.7; 20% edge cars included
R03.08.01.01-T02 Polygon around a lake (satellite)
Original instruction
In Label Studio, open sat_lake.png and draw a polygon labeled 'water' that traces the lake shoreline, following the inlets. Submit the task.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgmentclick· Point & click- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Polygon-to-gold IoU >= 0.85 for the lake region. Graded on the exported polygon; in-tool vertex placement required. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
50% lake covered (IoU>=0.7); 50% tight shoreline (IoU>=0.85)
R03.08.01.01-T03 Instance mask for a product (labelme)
Original instruction
In labelme, open bottle.jpg and draw a polygon mask labeled 'bottle' around the single product, excluding the shadow. Save the .json next to the image.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- labelme
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
The saved labelme polygon has IoU >= 0.85 to the gold mask and excludes the shadow region. Graded on the .json geometry. The desktop has no terminal or Python interpreter (labelme runs as a frozen executable), and a file-access audit must show labelme as the only process that wrote the .json.
Proposed partial credit
60% object masked (IoU>=0.75); 40% shadow excluded (IoU>=0.85)
R03.08.01.01-T04 Rotated boxes for document text regions
Original instruction
In CVAT, annotate scan_doc1.png and scan_doc2.png with rotated bounding boxes labeled 'textline' around each slanted line of text, matching the line orientation. Export the rotated-box annotations.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Rotated boxes match gold with mean IoU >= 0.7 and angle error <= 8 degrees per line; recall >= 0.9. Graded on the exported rotated boxes (in-tool rotation handles used). CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
40% all lines boxed; 30% IoU >= 0.7; 30% angle error <= 8 deg
R03.08.01.01-T05 Human-pose keypoints on synthetic figures
Original instruction
In CVAT, label the 17 COCO-style body keypoints on each of the 8 synthetic rendered figures in project 'pose8', marking occluded joints with the visibility flag rather than skipping them.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Mean keypoint position error (OKS) >= 0.8 vs gold and visibility flags match >= 90%. Graded on the exported keypoints. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
30% all joints placed; 40% OKS >= 0.8; 30% visibility flags correct
R03.08.01.01-T06 Fix five mislabeled boxes
Original instruction
In CVAT project 'review5', the review notes list 5 boxes labeled 'truck' that should be 'bus'. Change exactly those 5 labels and leave all other annotations untouched. Re-export.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
The 5 flagged boxes are relabeled bus and the class diff vs the original is exactly those 5; no geometry changed. Graded on the label diff. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
60% all 5 relabeled; 40% nothing else changed
R03.08.01.01-T07 Add occlusion attribute to hidden objects
Original instruction
In Label Studio project 'occ', every person box exists but the 'occluded' attribute is unset. Set occluded=true on every person who is partially hidden by another object and leave fully visible people as false. Submit all tasks.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
The occluded attribute matches gold for >= 90% of person instances. Graded on the attribute diff vs gold. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
50% occluded set where hidden; 50% visible left false (at least 90 percent overall)
R03.08.01.01-T08 Instance polygons for synthetic cells
Original instruction
In CVAT, open the synthetic microscopy image micro_01.png and draw a separate polygon labeled 'cell' around each cell instance, splitting touching cells into distinct instances. Export instance polygons.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgmentgeom· Precise geometry- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Instance mask AP (IoU 0.5) >= 0.75 vs gold, with the correct instance count (+/-2) and touching cells separated. Graded on exported instance polygons. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
30% cells detected; 30% instances separated; 40% mask IoU/AP threshold
R03.08.01.01-T09 Dense retail shelf product boxes
Original instruction
In CVAT, annotate shelf.jpg with a bounding box labeled 'product' around every item on the shelf (densely packed, many overlapping). Aim for complete recall on the front-facing products.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Recall >= 0.85 and mean IoU >= 0.65 for front-facing products vs gold. Graded on the exported boxes; in-tool drawing required. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
50% product recall; 30% IoU threshold; 20% no duplicate boxes
R03.08.01.01-T10 Semantic road segmentation mask
Original instruction
In CVAT (or Label Studio brush), paint a semantic mask on street_seg.png with classes 'road', 'sidewalk', and 'background', covering the whole image with no unlabeled pixels. Export the segmentation mask.
- Platforms
web· Any OS (web)- Applications
- CVAT · Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Per-class mean IoU >= 0.8 vs gold and <1% of pixels left unlabeled. Graded on the exported mask. Neither CVAT nor Label Studio issues the agent an API token and annotation import is disabled in both, so the mask can be produced only by painting in the browser UI.
Proposed partial credit
30% road IoU; 30% sidewalk IoU; 40% full coverage (no unlabeled)
R03.08.01.01-T11 Multi-class vehicle and person boxes
Original instruction
In CVAT project 'multi12' (12 images), box every instance of classes car, bus, person, and bicycle with the correct label. Export COCO.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Per-class recall >= 0.85 and mean IoU >= 0.7 vs gold; class confusion < 5%. Graded on the exported COCO. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
40% per-class recall; 40% IoU threshold; 20% low class confusion
R03.08.01.01-T12 Label small traffic signs with zoom
Original instruction
In CVAT, annotate highway.jpg with tight boxes labeled 'sign' around every traffic sign, including distant ones that require zooming in. Small signs must still be tightly bounded.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometrynav· Pan/zoom/scrollvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Recall >= 0.9 including small (<32px) signs; mean IoU >= 0.6 for small and >= 0.75 overall vs gold. Graded on exported boxes. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
40% overall recall; 30% small-sign recall; 30% IoU thresholds
R03.08.01.01-T13 Camera-trap animal boxes
Original instruction
In Label Studio, box every animal in the 15 camera-trap images of project 'wild15' with label 'animal', including animals that are motion-blurred or at the frame edge. Skip empty images (mark as reviewed).
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Recall >= 0.85 and mean IoU >= 0.6 vs gold; empty images correctly left with no boxes. Graded on exported boxes. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
40% animal recall; 30% IoU threshold; 30% empty images handled
R03.08.01.01-T14 Irregular boundary polygons (leaves)
Original instruction
In labelme, open leaves3.jpg and draw a polygon mask labeled 'leaf' around each of the three leaves, following the serrated edges closely. Save the .json.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- labelme
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Each leaf polygon has IoU >= 0.8 to gold with boundary adherence (mean contour distance below threshold). Graded on the saved polygons. The desktop has no terminal or Python interpreter (labelme runs as a frozen executable), and a file-access audit must show labelme as the only process that wrote the .json.
Proposed partial credit
40% three leaves masked; 30% IoU >= 0.75; 30% edge adherence
R03.08.01.01-T15 Tighten loose boxes
Original instruction
In CVAT project 'tighten', 12 car boxes are drawn too loosely. Resize each so it tightly bounds the car without clipping it. Do not add or remove boxes.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
After editing, mean IoU to gold rises to >= 0.85 for the 12 boxes and the box count is unchanged. Graded on the IoU improvement. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
50% all boxes tightened; 50% mean IoU >= 0.85
R03.08.01.01-T16 Brush defect regions on synthetic parts
Original instruction
In CVAT, paint a brush mask labeled 'defect' over the scratch and dent regions on part_01.png and part_02.png. Leave clean surface unlabeled. Export masks.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Defect-mask IoU >= 0.7 vs gold on both images with precision >= 0.8 (little over-paint onto clean areas). Graded on the exported masks. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
50% defects covered (IoU); 50% precision (no over-paint)
R03.08.01.01-T17 Overlapping instances with distinct IDs
Original instruction
In CVAT, annotate crowd_small.jpg where several people overlap; draw a separate box per person with a unique instance id even where bodies occlude each other, labeling the visible extent of each. Export.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Instance count matches gold (+/-1), each person has a distinct id, and mean IoU >= 0.6 for visible extents. Graded on exported instances. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
40% correct instance count; 30% distinct ids; 30% IoU threshold
R03.08.01.01-T18 Count-focused dense crowd heads
Original instruction
In CVAT, place a tight box labeled 'head' on every visible head in stadium.jpg (a dense crowd). Maximize recall; aim for complete coverage of the lower-and-mid stands where heads are resolvable.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometrynav· Pan/zoom/scrollvis· Visual judgment- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Recall >= 0.8 over resolvable heads vs gold with precision >= 0.85 (few false boxes) and count within 10% of gold. Graded on exported boxes. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
40% head recall; 30% count within 10 percent of gold; 30% precision
R03.08.01.01-T19 Organ boundary segmentation to Dice
Original instruction
In CVAT, trace a polygon labeled 'organ' around the target structure in each of the 6 synthetic cross-section images of project 'organ6', following the boundary precisely. Export masks.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Mean Dice >= 0.9 vs gold across the 6 images with boundary (Hausdorff) error below threshold. Graded on the exported masks. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
40% organ segmented on all images; 30% Dice >= 0.85; 30% Dice >= 0.9 + boundary
R03.08.01.01-T20 Combined boxes and masks to a joint threshold
Original instruction
In CVAT project 'joint8' (8 urban images), produce boxes for classes car/person and instance polygon masks for class 'building', all with correct labels. Export a COCO file containing both box and segmentation annotations.
- Platforms
web· Any OS (web)- Applications
- CVAT
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & droppath· Continuous pathgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Box recall >= 0.85 and IoU >= 0.7 for car/person, and building mask AP(0.5) >= 0.75 vs gold; the single export contains both annotation types. Graded on the combined COCO. CVAT accepts only browser sessions (CSRF + session cookie, no API token), annotation import and auto-annotation are disabled for the agent role, and the client event log must show the shapes drawn in the UI.
Proposed partial credit
35% car/person boxes meet threshold; 35% building masks meet AP; 30% single valid combined export
Mapped research references
No benchmark reference is assigned to this class in the current mapping.