Legal classification & reasoning
Classify documents and answer doctrinal questions (control class).
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- text tools
- Class GUI-need label
AP· API-sufficient
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R10.02.01.03-T01 Label thirty fact patterns as hearsay or not hearsay
Original instruction
input/hearsay_items.jsonl has 30 short fact patterns (synthetic), each describing an out-of-court statement and the purpose for which it is offered at trial under the Federal Rules of Evidence. Label each 'hearsay' or 'not hearsay' (ignore exceptions) and write output/labels.jsonl with id and label.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 1 / 5
Proposed verifier
Accuracy vs expert gold labels; pass >= 0.9. Items include effect-on-listener, verbal acts and party-opponent statements so surface cues do not decide the label.
Proposed partial credit
50% accuracy scaled linearly from 0.6 to 0.8; 50% accuracy scaled linearly from 0.8 to 1.0.
R10.02.01.03-T02 Decide diversity jurisdiction for twenty-five scenarios
Original instruction
input/diversity.jsonl: 25 synthetic lawsuits listing each party's citizenship facts (domicile, corporate state of incorporation and principal place of business, LLC member citizenship) and the amount claimed. For each, decide whether 28 U.S.C. 1332(a) diversity jurisdiction exists and give the failing element if not ('amount', 'complete diversity', or none).
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Label accuracy and failing-element accuracy vs gold; pass >= 0.92 on both. Includes LLC look-through and exactly-$75,000 cases.
Proposed partial credit
60% jurisdiction label; 40% failing element.
R10.02.01.03-T03 Tag twelve clause categories with evidence spans across ten contracts
Original instruction
input/contracts/ has 10 synthetic commercial agreements (text). For each contract, identify whether each of 12 categories in input/categories.md is present (e.g. non-compete, exclusivity, change of control, uncapped liability, audit rights, most-favored nation) and return the clause span offsets as evidence in output/tags.json.
- Platforms
lin· Linux- Applications
- text tools · Python
- Interfaces
api· API/SDKterm· Terminal- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Presence F1 per category vs gold (macro-F1 >= 0.85); evidence span character-level overlap with gold spans >= 0.5 IoU for true positives.
Proposed partial credit
60% presence macro-F1; 40% evidence-span IoU.
R10.02.01.03-T04 Select the correct holding for forty cited-case excerpts
Original instruction
input/holdings.jsonl has 40 items: a passage from a judicial opinion with a citation whose holding parenthetical is masked, and 5 candidate holding statements. Choose the candidate that correctly states the cited case's holding as used in context. Output output/answers.jsonl.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Accuracy vs gold; pass >= 0.8. Distractors are near-paraphrases with flipped outcomes, so keyword overlap does not decide.
Proposed partial credit
50% accuracy scaled linearly from 0.4 to 0.65; 50% accuracy scaled linearly from 0.65 to 0.9.
R10.02.01.03-T05 Classify AI-governance documents by status, scope and lifecycle with verbatim evidence
Original instruction
input/governance_docs/ has 6 synthetic AI-policy documents (bills, enacted acts, agency guidance). For each, assign legislative status (proposed/enacted/guidance), scope (sector list from input/scopes.md) and lifecycle stage(s) addressed (design, deployment, monitoring), each supported by a verbatim quote of at most 40 words. Write output/classification.json.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Labels vs gold (status exact; scope and lifecycle Jaccard >= 0.8); each quote must appear verbatim in the source (exact substring) and lie within a gold evidence region, otherwise its label scores 0 (evidence gate).
Proposed partial credit
30% status; 35% scope and lifecycle; 35% evidence gate passed.
R10.02.01.03-T06 Classify two thousand privacy-policy segments with a scripted pipeline
Original instruction
input/policy_segments.csv has 2,000 segments from 40 synthetic privacy policies. Build a script that labels each with one or more of 10 practice categories in input/taxonomy.md (first-party collection, third-party sharing, retention, user choice, security, etc.) and writes output/segment_labels.csv. Any model or rule approach is allowed; runtime under 30 minutes.
- Platforms
lin· Linux- Applications
- Python · LLM API
- Interfaces
term· Terminalapi· API/SDK- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Micro-F1 vs gold multi-labels >= 0.8 and macro-F1 >= 0.7; script reruns from scratch to the same output; runtime logged.
Proposed partial credit
60% micro-F1; 25% macro-F1; 15% reproducible run.
R10.02.01.03-T07 Apply the statute of frauds to twenty agreements
Original instruction
input/sof.jsonl: 20 synthetic oral or partly written agreements. For each, decide whether a signed writing is required under the common-law statute of frauds and UCC 2-201, name the category (land, suretyship, marriage, one-year, goods >= $500, none) and whether an exception (part performance, merchant confirmation, admission) saves it.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Per item: writing-required label, category and exception exact vs gold; pass if mean field accuracy >= 0.9.
Proposed partial credit
40% writing-required; 35% category; 25% exception.
R10.02.01.03-T08 Spot issues in a torts exam hypothetical from a closed issue list
Original instruction
input/torts_hypo.txt (2,000 words, synthetic) and input/issue_list.md (15 issue labels such as battery, negligence per se, res ipsa, comparative fault, NIED). Return output/issues.json listing each issue raised, the party pair it concerns, and a one-line conclusion (likely liable / not liable / disputed).
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Issue set F1 vs gold on (issue, party pair) >= 0.8; conclusion accuracy on matched issues >= 0.75. No free-text grading.
Proposed partial credit
60% issue F1; 40% conclusions.
R10.02.01.03-T09 Apply the Howey test to fifteen investment scenarios
Original instruction
input/howey.jsonl: 15 synthetic arrangements (tokens, pooled farms, profit-sharing memberships). For each decide whether it is an investment contract under the Howey test and, if not, name the first failing prong (investment of money, common enterprise, expectation of profits, efforts of others). Write output/howey.jsonl.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Label and failing-prong accuracy vs gold; pass >= 0.87 on labels.
Proposed partial credit
60% labels; 40% failing prong.
R10.02.01.03-T10 Judge seventeen hypotheses against each of ten NDAs with evidence
Original instruction
input/ndas/ has 10 synthetic NDAs and input/hypotheses.md lists 17 hypotheses (e.g. 'Receiving party may share with employees', 'Agreement survives termination'). For each NDA x hypothesis return entailed / contradicted / not mentioned with the supporting sentence indices. Write output/nli.json.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Label accuracy over 170 pairs >= 0.85; evidence sentence F1 >= 0.7 on entailed/contradicted pairs.
Proposed partial credit
65% labels; 35% evidence sentences.
R10.02.01.03-T11 Tag the same contract clauses by highlighting spans in Label Studio (GUI pair)
Original instruction
GUI pair of T03. The same 10 contracts are loaded as tasks in a Label Studio project with the 12 clause labels. Highlight each clause span in the browser with the right label and submit every task. The project's API token is not available to you.
- Platforms
web· Any OS (web)- Applications
- Label Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clicknav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Label Studio export: same metrics as T03 (presence macro-F1, span IoU). Personal and legacy API tokens are disabled in the organization settings, so annotations must be submitted from the browser session and carry that user's completed_by and lead_time > 0.
Proposed partial credit
60% presence macro-F1; 40% span IoU.
R10.02.01.03-T12 Answer ten multi-hop questions over filings and judgments with retrieval tools
Original instruction
A tool server exposes search_companies, get_filing, search_judgments and get_judgment over a synthetic corpus (300 company filings, 500 court judgments). Answer the 10 questions in input/questions.jsonl (e.g. which subsidiary of the defendant in judgment X was later held jointly liable, and under which statute) with exact entity names, dates and amounts.
- Platforms
lin· Linux- Applications
- MCP tool server · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Exact match on normalized answers per question; progress credit from the gold hop chain (fraction of required documents retrieved, from the tool-call log).
Proposed partial credit
70% final answers; 30% hop-chain progress.
R10.02.01.03-T13 Classify workers under the ABC test and name the failing prong
Original instruction
input/workers.jsonl: 30 synthetic working arrangements in California. For each, classify employee vs independent contractor under the ABC test (Lab. Code 2775), name the first prong the hiring entity fails if employee, and flag the 4 categories exempted by the statute (exemption name) where applicable.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Classification accuracy >= 0.9; failing-prong accuracy on employee items; exemption flags exact on the seeded exempt items.
Proposed partial credit
50% classification; 30% prong; 20% exemptions.
R10.02.01.03-T14 Route forty client intake questions to practice areas
Original instruction
input/intake.jsonl: 40 plain-language client questions (synthetic). Assign each one primary practice area from input/areas.md (12 options such as landlord-tenant, family, employment, immigration, consumer debt) for a legal-aid intake queue. Output output/routing.jsonl.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Accuracy vs gold >= 0.9; confusions between adjacent areas reported but not separately weighted.
Proposed partial credit
50% accuracy scaled linearly from 0.6 to 0.775; 50% accuracy scaled linearly from 0.775 to 0.95.
R10.02.01.03-T15 Make privilege calls on thirty email summaries with reason codes
Original instruction
input/emails.jsonl: 30 synthetic email summaries with sender, recipients (roles given) and content gist. Label each attorney-client privileged, work product, both or none, and give a reason code from input/reasons.md (e.g. third-party waiver, business advice, in anticipation of litigation).
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Label accuracy >= 0.85 and reason-code accuracy >= 0.75 vs gold; seeded waiver cases (outside consultant cc'd) must be 'none'.
Proposed partial credit
55% labels; 30% reason codes; 15% waiver cases.
R10.02.01.03-T16 Compute arbitration administrative fees and arbitrator fee ranges from a scale
Original instruction
input/fee_scale.pdf is a synthetic arbitral institution's schedule (administrative fee and arbitrator fee min/max by amount-in-dispute band, sole vs three-member tribunal rules). For the 6 disputes in input/disputes.json (some in EUR, convert at input/fx.json), compute administrative fee, arbitrator fee min and max, and total maximum advance in USD.
- Platforms
lin· Linux- Applications
- text tools · Python
- Interfaces
api· API/SDKterm· Terminal- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
24 numbers vs gold within 0.5 USD; band-boundary disputes seeded; tribunal multiplier applied only where the rules say.
Proposed partial credit
35% administrative fees; 45% arbitrator fee ranges; 20% totals.
R10.02.01.03-T17 Classify governance documents shown only as page images in a viewer (GUI pair)
Original instruction
GUI pair of T05. The same 6 AI-policy documents are available only in a browser document viewer that renders page images (no text layer, no download, copy disabled). Classify status, scope and lifecycle and give a verbatim quote of at most 40 words for each label, typed from what you read. Write output/classification.json.
- Platforms
web· Any OS (web)- Applications
- browser document viewer
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentnav· Pan/zoom/scrolltype· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Same scoring as T05; quotes matched against source text with <= 2 character edits allowed per quote. The viewer serves page images only to its browser session (no download or text endpoint) and OCR tools are absent from the VM; viewer logs confirm pages were viewed. Measures reading-through-GUI cost for identical reasoning.
Proposed partial credit
30% status; 35% scope and lifecycle; 35% evidence gate passed.
R10.02.01.03-T18 Assign EU AI Act risk categories to twenty-five product descriptions
Original instruction
input/products.jsonl: 25 synthetic AI product descriptions. For each, assign the EU AI Act category (prohibited practice, high-risk under Annex I or Annex III, transparency obligations only, or minimal) and cite the specific article or Annex III point relied on. Write output/ai_act.jsonl.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Category accuracy >= 0.85; cited article/annex point exact on correctly categorized items; seeded edge cases (Article 6(3) derogation) included.
Proposed partial credit
60% categories; 40% citations.
R10.02.01.03-T19 Score sixty clauses against an explicit negotiation playbook
Original instruction
input/clauses.jsonl has 60 clauses from synthetic vendor contracts and input/playbook.md gives explicit rules (e.g. cap below 12 months' fees is unacceptable; auto-renewal with < 60 days notice needs negotiation). Label each clause acceptable / needs negotiation / unacceptable and name the rule ID applied.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Label accuracy >= 0.9 and rule-ID accuracy >= 0.85 vs gold derived mechanically from the playbook.
Proposed partial credit
60% labels; 40% rule IDs.
R10.02.01.03-T20 Answer doctrinal questions over a long case record with paragraph citations
Original instruction
input/record/ is a 200-page synthetic case record (complaint, declarations, deposition excerpts, prior orders) with numbered paragraphs. Answer the 15 questions in input/questions.md (e.g. whether specific personal jurisdiction elements are met, whether a claim is time-barred) with a fixed label each and the record paragraph IDs supporting it.
- Platforms
lin· Linux- Applications
- text tools · LLM API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Label accuracy vs gold >= 0.8; cited paragraph IDs must include at least one gold support paragraph per answer (else that answer scores half).
Proposed partial credit
65% labels; 35% supporting citations.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.