R10.02.01.03 · 任务小类

法律分类与推理

Legal classification & reasoning

Classify documents and answer doctrinal questions (control class).

20 条本项目编写的提案 · 分类 v1.1 · 新增基准运行 0 次。下方验证方式与部分得分规则是设计，不是执行结果。

典型应用: text tools
小类 GUI 需求标签: AP · API 足够

任务提案 · 20

展开查看完整原文与建议检查方式；两种界面均保留提案英文原文。

R10.02.01.03-T01 Label thirty fact patterns as hearsay or not hearsay

原始指令

input/hearsay_items.jsonl has 30 short fact patterns (synthetic), each describing an out-of-court statement and the purpose for which it is offered at trial under the Federal Rules of Evidence. Label each 'hearsay' or 'not hearsay' (ignore exceptions) and write output/labels.jsonl with id and label.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 1 / 5

建议验证方式

Accuracy vs expert gold labels; pass >= 0.9. Items include effect-on-listener, verbal acts and party-opponent statements so surface cues do not decide the label.

建议部分得分

50% accuracy scaled linearly from 0.6 to 0.8; 50% accuracy scaled linearly from 0.8 to 1.0.

R10.02.01.03-T02 Decide diversity jurisdiction for twenty-five scenarios

原始指令

input/diversity.jsonl: 25 synthetic lawsuits listing each party's citizenship facts (domicile, corporate state of incorporation and principal place of business, LLC member citizenship) and the amount claimed. For each, decide whether 28 U.S.C. 1332(a) diversity jurisdiction exists and give the failing element if not ('amount', 'complete diversity', or none).

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 2 / 5

建议验证方式

Label accuracy and failing-element accuracy vs gold; pass >= 0.92 on both. Includes LLC look-through and exactly-$75,000 cases.

建议部分得分

60% jurisdiction label; 40% failing element.

R10.02.01.03-T03 Tag twelve clause categories with evidence spans across ten contracts

原始指令

input/contracts/ has 10 synthetic commercial agreements (text). For each contract, identify whether each of 12 categories in input/categories.md is present (e.g. non-compete, exclusivity, change of control, uncapped liability, audit rights, most-favored nation) and return the clause span offsets as evidence in output/tags.json.

平台: lin · Linux
应用: text tools · Python
交互表面: api · API / SDK term · 终端
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP10 · 标注
条件: 未声明
提案难度: 3 / 5

建议验证方式

Presence F1 per category vs gold (macro-F1 >= 0.85); evidence span character-level overlap with gold spans >= 0.5 IoU for true positives.

建议部分得分

60% presence macro-F1; 40% evidence-span IoU.

R10.02.01.03-T04 Select the correct holding for forty cited-case excerpts

原始指令

input/holdings.jsonl has 40 items: a passage from a judicial opinion with a citation whose holding parenthetical is masked, and 5 candidate holding statements. Choose the candidate that correctly states the cited case's holding as used in context. Output output/answers.jsonl.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Accuracy vs gold; pass >= 0.8. Distractors are near-paraphrases with flipped outcomes, so keyword overlap does not decide.

建议部分得分

50% accuracy scaled linearly from 0.4 to 0.65; 50% accuracy scaled linearly from 0.65 to 0.9.

R10.02.01.03-T05 Classify AI-governance documents by status, scope and lifecycle with verbatim evidence

原始指令

input/governance_docs/ has 6 synthetic AI-policy documents (bills, enacted acts, agency guidance). For each, assign legislative status (proposed/enacted/guidance), scope (sector list from input/scopes.md) and lifecycle stage(s) addressed (design, deployment, monitoring), each supported by a verbatim quote of at most 40 words. Write output/classification.json.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 4 / 5

建议验证方式

Labels vs gold (status exact; scope and lifecycle Jaccard >= 0.8); each quote must appear verbatim in the source (exact substring) and lie within a gold evidence region, otherwise its label scores 0 (evidence gate).

建议部分得分

30% status; 35% scope and lifecycle; 35% evidence gate passed.

R10.02.01.03-T06 Classify two thousand privacy-policy segments with a scripted pipeline

原始指令

input/policy_segments.csv has 2,000 segments from 40 synthetic privacy policies. Build a script that labels each with one or more of 10 practice categories in input/taxonomy.md (first-party collection, third-party sharing, retention, user choice, security, etc.) and writes output/segment_labels.csv. Any model or rule approach is allowed; runtime under 30 minutes.

平台: lin · Linux
应用: Python · LLM API
交互表面: term · 终端 api · API / SDK
GUI 需求 · 任务设定: CL · CLI 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP10 · 标注
条件: 未声明
提案难度: 3 / 5

建议验证方式

Micro-F1 vs gold multi-labels >= 0.8 and macro-F1 >= 0.7; script reruns from scratch to the same output; runtime logged.

建议部分得分

60% micro-F1; 25% macro-F1; 15% reproducible run.

R10.02.01.03-T07 Apply the statute of frauds to twenty agreements

原始指令

input/sof.jsonl: 20 synthetic oral or partly written agreements. For each, decide whether a signed writing is required under the common-law statute of frauds and UCC 2-201, name the category (land, suretyship, marriage, one-year, goods >= $500, none) and whether an exception (part performance, merchant confirmation, admission) saves it.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Per item: writing-required label, category and exception exact vs gold; pass if mean field accuracy >= 0.9.

建议部分得分

40% writing-required; 35% category; 25% exception.

R10.02.01.03-T08 Spot issues in a torts exam hypothetical from a closed issue list

原始指令

input/torts_hypo.txt (2,000 words, synthetic) and input/issue_list.md (15 issue labels such as battery, negligence per se, res ipsa, comparative fault, NIED). Return output/issues.json listing each issue raised, the party pair it concerns, and a one-line conclusion (likely liable / not liable / disputed).

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 4 / 5

建议验证方式

Issue set F1 vs gold on (issue, party pair) >= 0.8; conclusion accuracy on matched issues >= 0.75. No free-text grading.

建议部分得分

60% issue F1; 40% conclusions.

R10.02.01.03-T09 Apply the Howey test to fifteen investment scenarios

原始指令

input/howey.jsonl: 15 synthetic arrangements (tokens, pooled farms, profit-sharing memberships). For each decide whether it is an investment contract under the Howey test and, if not, name the first failing prong (investment of money, common enterprise, expectation of profits, efforts of others). Write output/howey.jsonl.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Label and failing-prong accuracy vs gold; pass >= 0.87 on labels.

建议部分得分

60% labels; 40% failing prong.

R10.02.01.03-T10 Judge seventeen hypotheses against each of ten NDAs with evidence

原始指令

input/ndas/ has 10 synthetic NDAs and input/hypotheses.md lists 17 hypotheses (e.g. 'Receiving party may share with employees', 'Agreement survives termination'). For each NDA x hypothesis return entailed / contradicted / not mentioned with the supporting sentence indices. Write output/nli.json.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 4 / 5

建议验证方式

Label accuracy over 170 pairs >= 0.85; evidence sentence F1 >= 0.7 on entailed/contradicted pairs.

建议部分得分

65% labels; 35% evidence sentences.

R10.02.01.03-T11 Tag the same contract clauses by highlighting spans in Label Studio (GUI pair)

原始指令

GUI pair of T03. The same 10 contracts are loaded as tasks in a Label Studio project with the 12 clause labels. Highlight each clause span in the browser with the right label and submit every task. The project's API token is not available to you.

平台: web · 任意系统（网页）
应用: Label Studio
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: path · 连续路径 click · 指向与点击 nav · 平移 / 缩放 / 滚动
工作产物: WP10 · 标注
条件: 未声明
提案难度: 3 / 5

建议验证方式

Label Studio export: same metrics as T03 (presence macro-F1, span IoU). Personal and legacy API tokens are disabled in the organization settings, so annotations must be submitted from the browser session and carry that user's completed_by and lead_time > 0.

建议部分得分

60% presence macro-F1; 40% span IoU.

R10.02.01.03-T12 Answer ten multi-hop questions over filings and judgments with retrieval tools

原始指令

A tool server exposes search_companies, get_filing, search_judgments and get_judgment over a synthetic corpus (300 company filings, 500 court judgments). Answer the 10 questions in input/questions.jsonl (e.g. which subsidiary of the defendant in judgment X was later held jointly liable, and under which statute) with exact entity names, dates and amounts.

平台: lin · Linux
应用: MCP tool server · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: AP · API 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: long · 长流程
提案难度: 5 / 5

建议验证方式

Exact match on normalized answers per question; progress credit from the gold hop chain (fraction of required documents retrieved, from the tool-call log).

建议部分得分

70% final answers; 30% hop-chain progress.

R10.02.01.03-T13 Classify workers under the ABC test and name the failing prong

原始指令

input/workers.jsonl: 30 synthetic working arrangements in California. For each, classify employee vs independent contractor under the ABC test (Lab. Code 2775), name the first prong the hiring entity fails if employee, and flag the 4 categories exempted by the statute (exemption name) where applicable.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Classification accuracy >= 0.9; failing-prong accuracy on employee items; exemption flags exact on the seeded exempt items.

建议部分得分

50% classification; 30% prong; 20% exemptions.

R10.02.01.03-T14 Route forty client intake questions to practice areas

原始指令

input/intake.jsonl: 40 plain-language client questions (synthetic). Assign each one primary practice area from input/areas.md (12 options such as landlord-tenant, family, employment, immigration, consumer debt) for a legal-aid intake queue. Output output/routing.jsonl.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 2 / 5

建议验证方式

Accuracy vs gold >= 0.9; confusions between adjacent areas reported but not separately weighted.

建议部分得分

50% accuracy scaled linearly from 0.6 to 0.775; 50% accuracy scaled linearly from 0.775 to 0.95.

R10.02.01.03-T15 Make privilege calls on thirty email summaries with reason codes

原始指令

input/emails.jsonl: 30 synthetic email summaries with sender, recipients (roles given) and content gist. Label each attorney-client privileged, work product, both or none, and give a reason code from input/reasons.md (e.g. third-party waiver, business advice, in anticipation of litigation).

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Label accuracy >= 0.85 and reason-code accuracy >= 0.75 vs gold; seeded waiver cases (outside consultant cc'd) must be 'none'.

建议部分得分

55% labels; 30% reason codes; 15% waiver cases.

R10.02.01.03-T16 Compute arbitration administrative fees and arbitrator fee ranges from a scale

原始指令

input/fee_scale.pdf is a synthetic arbitral institution's schedule (administrative fee and arbitrator fee min/max by amount-in-dispute band, sole vs three-member tribunal rules). For the 6 disputes in input/disputes.json (some in EUR, convert at input/fx.json), compute administrative fee, arbitrator fee min and max, and total maximum advance in USD.

平台: lin · Linux
应用: text tools · Python
交互表面: api · API / SDK term · 终端
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 4 / 5

建议验证方式

24 numbers vs gold within 0.5 USD; band-boundary disputes seeded; tribunal multiplier applied only where the rules say.

建议部分得分

35% administrative fees; 45% arbitrator fee ranges; 20% totals.

R10.02.01.03-T17 Classify governance documents shown only as page images in a viewer (GUI pair)

原始指令

GUI pair of T05. The same 6 AI-policy documents are available only in a browser document viewer that renders page images (no text layer, no download, copy disabled). Classify status, scope and lifecycle and give a verbatim quote of at most 40 words for each label, typed from what you read. Write output/classification.json.

平台: web · 任意系统（网页）
应用: browser document viewer
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G2 · GUI 必要
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: vis · 视觉判断 nav · 平移 / 缩放 / 滚动 type · 文字与表单
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 5 / 5

建议验证方式

Same scoring as T05; quotes matched against source text with <= 2 character edits allowed per quote. The viewer serves page images only to its browser session (no download or text endpoint) and OCR tools are absent from the VM; viewer logs confirm pages were viewed. Measures reading-through-GUI cost for identical reasoning.

建议部分得分

30% status; 35% scope and lifecycle; 35% evidence gate passed.

R10.02.01.03-T18 Assign EU AI Act risk categories to twenty-five product descriptions

原始指令

input/products.jsonl: 25 synthetic AI product descriptions. For each, assign the EU AI Act category (prohibited practice, high-risk under Annex I or Annex III, transparency obligations only, or minimal) and cite the specific article or Annex III point relied on. Write output/ai_act.jsonl.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 4 / 5

建议验证方式

Category accuracy >= 0.85; cited article/annex point exact on correctly categorized items; seeded edge cases (Article 6(3) derogation) included.

建议部分得分

60% categories; 40% citations.

R10.02.01.03-T19 Score sixty clauses against an explicit negotiation playbook

原始指令

input/clauses.jsonl has 60 clauses from synthetic vendor contracts and input/playbook.md gives explicit rules (e.g. cap below 12 months' fees is unacceptable; auto-renewal with < 60 days notice needs negotiation). Label each clause acceptable / needs negotiation / unacceptable and name the rule ID applied.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: 未声明
提案难度: 3 / 5

建议验证方式

Label accuracy >= 0.9 and rule-ID accuracy >= 0.85 vs gold derived mechanically from the playbook.

建议部分得分

60% labels; 40% rule IDs.

R10.02.01.03-T20 Answer doctrinal questions over a long case record with paragraph citations

原始指令

input/record/ is a 200-page synthetic case record (complaint, declarations, deposition excerpts, prior orders) with numbered paragraphs. Answer the 15 questions in input/questions.md (e.g. whether specific personal jurisdiction elements are met, whether a claim is time-barred) with a fixed label each and the record paragraph IDs supporting it.

平台: lin · Linux
应用: text tools · LLM API
交互表面: api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP12 · 答案 / 决策
条件: long · 长流程
提案难度: 5 / 5

建议验证方式

Label accuracy vs gold >= 0.8; cited paragraph IDs must include at least one gold support paragraph per answer (else that answer scores half).

建议部分得分

65% labels; 35% supporting citations.

已有研究映射

以下沿用研究目录中的基准与小类映射，不是逐题标注、执行记录或产品背书。

法律分类与推理

任务提案 · 20

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

已有研究映射

同工作流族的小类