Text, audio & document labeling
Label spans, segments and document fields consistently with a guideline.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Label Studio · Doccano · Prodigy
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R03.08.01.03-T01 Label PERSON and ORG spans
Original instruction
In Doccano project 'ner20' (20 news sentences), label every PERSON and ORG mention as a span with the correct type, following the guideline (titles excluded from PERSON; include full org names). Submit all documents.
- Platforms
web· Any OS (web)- Applications
- Doccano · Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Span-level F1 >= 0.9 vs gold for both PERSON and ORG (exact boundary + type). Graded on exported spans; labels must be created in the tool. Doccano issues the agent no API token and its annotator role cannot import datasets, so labels can be created only in the annotation UI.
Proposed partial credit
40% entities found; 40% F1 >= 0.9; 20% boundary rules followed
R03.08.01.03-T02 Classify 30 reviews by sentiment
Original instruction
In Label Studio project 'sent30', assign each of the 30 product reviews one label: positive, negative, or neutral, per the guideline (mixed reviews -> neutral). Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Classification accuracy >= 0.9 vs gold across the 30 reviews. Graded on the submitted labels. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
20% all 30 reviews labeled and submitted; 80% scaled by accuracy (pass at >=0.9)
R03.08.01.03-T03 Segment speaker turns in audio
Original instruction
In Label Studio audio (or Audacity label track), open convo.wav (2-minute two-speaker dialogue) and mark each speaker turn as a labeled region (SpeakerA/SpeakerB), covering the whole clip with no gaps during speech. Submit.
- Platforms
web· Any OS (web)win· Windowsmac· macOSlin· Linux- Applications
- Label Studio · Audacity
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropav· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Turn boundary error <= 0.3 s vs gold and speaker labels correct for >= 90% of speech time. Graded on the exported regions. Label Studio API tokens and import are disabled for the agent; in Audacity, mod-script-pipe and the Nyquist Prompt are disabled and a file-access audit must show Audacity wrote any label export.
Proposed partial credit
40% turns segmented; 30% boundary error <= 0.3s; 30% speaker labels correct
R03.08.01.03-T04 Annotate relations between entities
Original instruction
In Doccano relation mode, project 'rel15', entities are pre-labeled; add the relation links (works_for, located_in, founded_by) between the correct entity pairs per the guideline across 15 documents. Submit.
- Platforms
web· Any OS (web)- Applications
- Doccano · Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Relation-level F1 >= 0.85 vs gold (correct pair + type). Graded on exported relations. Doccano issues the agent no API token and its annotator role cannot import datasets, so labels can be created only in the annotation UI.
Proposed partial credit
40% relations added; 40% F1 >= 0.85; 20% no spurious links
R03.08.01.03-T05 Coreference clustering
Original instruction
In INCEpTION/Label Studio project 'coref10', link all mentions that refer to the same entity into coreference chains across each of the 10 short passages, following the guideline on pronouns and nominal mentions. Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Coreference CoNLL F1 (MUC/B3/CEAF average) >= 0.8 vs gold. Graded on the exported chains. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
30% mentions linked; 40% F1 >= 0.8; 30% pronoun rule followed
R03.08.01.03-T06 Label invoice fields on scans (KIE)
Original instruction
In Label Studio project 'invoice8' (8 scanned invoices), label the key fields as spans/regions: vendor name, invoice number, invoice date, total amount, and tax. Use the region tool on the scanned image text. Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Per-field accuracy >= 0.9 vs gold (correct region containing the right value) across the 8 invoices. Graded on the labeled regions. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
40% all fields present; 40% field accuracy >= 0.9; 20% region boundaries correct
R03.08.01.03-T07 Classify utterance intents
Original instruction
In Label Studio project 'intent40', assign each of the 40 chat utterances one intent from the 6-class scheme (greeting, order, cancel, refund, question, other) per the guideline. Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & click- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Intent accuracy >= 0.9 vs gold across the 40 utterances. Graded on the submitted labels. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
20% all 40 utterances labeled and submitted; 80% scaled by accuracy (pass at >=0.9)
R03.08.01.03-T08 Fine-grained NER with five classes
Original instruction
In Doccano project 'ner5class' (25 biomedical-style synthetic sentences), label spans for DISEASE, DRUG, DOSAGE, FREQUENCY, and SYMPTOM per the guideline (dosage includes units). Submit.
- Platforms
web· Any OS (web)- Applications
- Doccano
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Macro span F1 >= 0.85 across the five classes vs gold (exact span + type). Graded on exported spans. Doccano issues the agent no API token and its annotator role cannot import datasets, so labels can be created only in the annotation UI.
Proposed partial credit
30% entities found; 40% macro F1 >= 0.85; 30% unit-inclusion rule for dosage
R03.08.01.03-T09 Label non-speech audio events
Original instruction
In Label Studio audio project 'events', mark labeled regions for each cough, laugh, and music segment in clip.wav, leaving speech unlabeled. Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio · Audacity
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropav· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Event-level F1 >= 0.8 vs gold with onset error <= 0.3 s per event. Graded on exported regions. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
40% events detected; 30% onset error <= 0.3s; 30% F1 >= 0.8
R03.08.01.03-T10 Aspect-based sentiment spans
Original instruction
In Label Studio project 'absa20' (20 restaurant reviews), label each aspect term span and attach a sentiment (positive/negative/neutral) attribute to it, per the guideline. Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Aspect-span F1 >= 0.8 and sentiment accuracy on matched aspects >= 0.85 vs gold. Graded on exported spans + attributes. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
40% aspect spans; 30% span F1 >= 0.8; 30% sentiment accuracy
R03.08.01.03-T11 Label PII types in documents
Original instruction
In Doccano project 'pii12' (12 synthetic documents), label spans for the PII categories NAME, ADDRESS, PHONE, EMAIL, and ID_NUMBER per the guideline. All data is synthetic. Submit.
- Platforms
web· Any OS (web)- Applications
- Doccano · Prodigy
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Span F1 >= 0.9 per category vs gold. Graded on exported spans; synthetic data only. Doccano issues the agent no API token and its annotator role cannot import datasets, so labels can be created only in the annotation UI.
Proposed partial credit
40% PII found; 40% F1 >= 0.9; 20% category precision
R03.08.01.03-T12 Multi-label topic tagging
Original instruction
In Label Studio project 'topics25', assign each of the 25 articles all applicable topic tags from the 8-tag scheme (an article may have several), per the guideline. Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Micro-averaged multi-label F1 >= 0.85 vs gold across the 25 articles. Graded on the submitted tag sets. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
50% tags assigned; 50% micro F1 >= 0.85
R03.08.01.03-T13 Sequence labeling in Doccano
Original instruction
In Doccano sequence-labeling project 'chunk15' (15 sentences), label each token span with its chunk tag (NP, VP, PP) per the guideline so adjacent same-type tokens form one span. Submit.
- Platforms
web· Any OS (web)- Applications
- Doccano
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & click- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Chunk span F1 >= 0.85 vs gold (boundaries + type). Graded on exported spans. Doccano issues the agent no API token and its annotator role cannot import datasets, so labels can be created only in the annotation UI.
Proposed partial credit
40% tokens labeled; 40% F1 >= 0.85; 20% span merging correct
R03.08.01.03-T14 Label noise vs speech regions in Audacity
Original instruction
In Audacity, open noisy_clip.wav and add a label track marking regions as 'speech' or 'noise', so every segment with background noise interference is captured. Export the label track as labels.txt.
- Platforms
win· Windowsmac· macOSlin· Linux- Applications
- Audacity
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
drag· Drag & dropav· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
labels.txt region boundaries match gold within 0.3 s and the speech/noise labels agree >= 90% by time. Graded on the exported label track. Audacity's mod-script-pipe and Nyquist Prompt are disabled, the desktop has no terminal, and a file-access audit must show Audacity as the process that wrote labels.txt.
Proposed partial credit
40% regions marked; 30% boundary tolerance; 30% label agreement
R03.08.01.03-T15 Label document layout regions
Original instruction
In Label Studio project 'layout6' (6 scanned pages), draw region boxes labeled Title, Paragraph, Table, and Figure over each page element per the guideline (captions belong to Figure). Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Region-level mean IoU >= 0.7 and label accuracy >= 0.9 vs gold across the 6 pages. Graded on exported regions. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
40% all regions boxed; 30% IoU >= 0.7; 30% label accuracy
R03.08.01.03-T16 Binary toxicity classification
Original instruction
In Label Studio project 'tox30', label each of 30 forum comments toxic or non-toxic per the guideline (sarcasm without slurs -> non-toxic). Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Accuracy >= 0.9 and F1 on the toxic class >= 0.85 vs gold. Graded on the submitted labels. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
50% accuracy >=0.9; 50% toxic-class F1 >= 0.85
R03.08.01.03-T17 Nested entity annotation
Original instruction
In INCEpTION/Label Studio project 'nested10', label nested entities where one entity contains another (e.g. an ORG inside a longer FACILITY name), creating both the inner and outer spans per the guideline. Submit.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clickvis· Visual judgment- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Both inner and outer span F1 >= 0.85 vs gold, with nesting relationships preserved. Graded on the exported nested spans. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
30% outer spans; 30% inner spans; 40% nesting correct + F1 >= 0.85
R03.08.01.03-T18 Word-level audio alignment
Original instruction
In Label Studio audio (or Praat/Audacity label track), align each word of the provided transcript of speech.wav to its spoken interval, producing one labeled region per word. Submit.
- Platforms
web· Any OS (web)win· Windowsmac· macOSlin· Linux- Applications
- Label Studio · Audacity
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropav· Audio/video over timetype· Text & forms- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Per-word onset/offset error <= 0.1 s vs gold for >= 90% of words and word order preserved. Graded on the exported word regions. Label Studio API tokens and import are disabled for the agent; in Audacity, mod-script-pipe and the Nyquist Prompt are disabled and a file-access audit must show Audacity wrote any label export.
Proposed partial credit
40% all words placed in order; 30% onset error; 30% offset error
R03.08.01.03-T19 Guideline consistency on edge cases
Original instruction
In Doccano project 'edge20', 20 sentences contain the ambiguous cases enumerated in the guideline (e.g. nationalities, product-as-org, abbreviations). Apply the guideline's rules consistently to label entity spans across all of them. Submit.
- Platforms
web· Any OS (web)- Applications
- Doccano
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
clarify· Ask when unclear- Proposed difficulty
- 3 / 5
Proposed verifier
Span F1 >= 0.9 vs the guideline-derived gold, with the edge-case rules applied consistently (consistency check across the 20 items). Graded on exported spans + a rule-consistency metric. Doccano issues the agent no API token and its annotator role cannot import datasets, so labels can be created only in the annotation UI.
Proposed partial credit
40% entities found; 30% F1 >= 0.9; 30% edge-case consistency
R03.08.01.03-T20 Combined entities, relations, and fields
Original instruction
In Label Studio project 'mixed10' (10 contracts), label entity spans (party, date, amount), relations (party-signs-contract), and the document field 'effective_date' on each, following the guideline. Submit all tasks.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropclick· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Entity span F1 >= 0.85, relation F1 >= 0.8, and effective_date field accuracy >= 0.9 vs gold across the 10 contracts. Graded on the combined export. Label Studio API tokens are disabled for the agent and import is turned off, so annotations can be submitted only from the labeling UI; each must carry a non-zero lead_time.
Proposed partial credit
35% entity spans; 30% relations; 35% effective_date field
Mapped research references
No benchmark reference is assigned to this class in the current mapping.