R10.01.01.01 · 任务小类

电子取证审阅、编码与涂黑

eDiscovery review, coding & redaction

Review document sets, apply issue codes and privilege redactions.

20 条本项目编写的提案 · 分类 v1.1 · 新增基准运行 0 次。下方验证方式与部分得分规则是设计，不是执行结果。

典型应用: Relativity · Everlaw · DISCO
小类 GUI 需求标签: G3 · 仅 GUI

任务提案 · 20

展开查看完整原文与建议检查方式；两种界面均保留提案英文原文。

R10.01.01.01-T01 Code a 20-document batch for responsiveness in Relativity

原始指令

Relativity sandbox workspace 'Halvorsen v. Kestrel Freight (synthetic)'. Check out review batch FL-001 (20 emails) and code each Responsive or Not Responsive on the 'First Level' coding layout following input/review_protocol.pdf section 2 (responsive = discusses the 2024 fuel-surcharge agreement). Check the batch back in.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 nav · 平移 / 缩放 / 滚动
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 2 / 5

建议验证方式

Responsiveness field per document vs gold (accuracy >= 0.9); batch status checked in. REST API access and mass-edit permission are denied to the reviewer group, so Relativity History must show per-document edits from the coding layout.

建议部分得分

80% coding accuracy (scaled from 0.6 to 1.0); 20% batch checked in.

R10.01.01.01-T02 Review a 50-email batch for responsiveness and privilege

原始指令

Relativity sandbox, batch FL-007 (50 emails with attachments). Per input/review_protocol.pdf code each document Responsive/Not Responsive and, where applicable, Privileged with type (Attorney-Client, Work Product) and a one-line privilege basis. Outside counsel names are listed in input/privilege_names.csv.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单 nav · 平移 / 缩放 / 滚动
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 3 / 5

建议验证方式

Responsiveness F1 >= 0.85 and privilege F1 >= 0.85 vs gold; privilege type exact on true positives; basis text non-empty. Coding must come from the layout (History audit; no mass edit, no API).

建议部分得分

45% responsiveness F1; 40% privilege F1 and type; 15% privilege basis present.

R10.01.01.01-T03 Redact Social Security numbers across ten imaged documents

原始指令

Relativity sandbox: saved search 'PII-SSN-Imaged' returns 10 imaged documents (scanned HR forms and letters, synthetic). Using the image viewer, draw black-box redactions with the label 'PII' over every SSN, including handwritten ones, without covering adjacent names or dates.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: geom · 精确几何 vis · 视觉判断 click · 指向与点击
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 3 / 5

建议验证方式

Markup table: redaction boxes vs gold SSN boxes, each gold box covered with IoU >= 0.5 and >= 95% coverage; OCR of a test production shows no 9-digit SSN pattern; boxes covering gold 'keep' regions (names/dates) count as over-redaction (max 1). REST API access is denied to the reviewer group and automated (search-term) redaction is disabled, so History must show markups drawn in the image viewer.

建议部分得分

60% all SSNs covered; 20% labels correct; 20% no over-redaction.

R10.01.01.01-T04 Apply partial privilege redactions and log them

原始指令

Everlaw sandbox project: 12 responsive emails contain a paragraph quoting outside counsel's advice. Redact only those paragraphs with the 'Privilege - AC' stamp, code each document 'Produce with redactions', and fill the redaction-log fields (date, author, recipients, privilege basis) for each.

平台: web · 任意系统（网页）
应用: Everlaw
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: geom · 精确几何 vis · 视觉判断 type · 文字与表单
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 4 / 5

建议验证方式

Redaction export: per document, redacted region covers the gold privileged paragraph (coverage >= 95%) and leaves >= 90% of the non-privileged text visible; coding exact; redaction-log fields vs gold (dates/authors exact). Project API is disabled for the reviewer.

建议部分得分

50% redaction precision/recall; 20% coding; 30% log fields.

R10.01.01.01-T05 Build a persistent highlight set and a saved search for review batches

原始指令

Relativity sandbox: create a persistent highlight set 'Surcharge Terms' with the 6 terms in input/terms.txt (two with proximity: fuel w/5 surcharge, rebate w/3 tier) in distinct colors, then a saved search returning unreviewed documents hitting any term, and a batch set of 25-document batches from it.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: licensed · 授权专业软件
提案难度: 3 / 5

建议验证方式

Workspace objects: highlight set with 6 terms and syntax exact; saved search conditions return the gold document set (set equality); batch set with size 25 and batch count = ceil(hits/25). REST API and Relativity Scripts access are denied to the agent account, so the workspace audit must show each object created from the UI.

建议部分得分

35% highlight set; 40% saved search returns gold set; 25% batch set.

R10.01.01.01-T06 Code 120 documents with family consistency and resolve reviewer conflicts

原始指令

Relativity sandbox, batch set 'Families-QC': 120 documents in 34 families, partially coded by two reviewers with conflicts. Apply input/review_protocol.pdf family rules (a family is produced if any member is responsive; privileged members withheld with slip sheet), recode conflicting members, and set 'Family QC Complete' on each family parent.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 nav · 平移 / 缩放 / 滚动 vis · 视觉判断
工作产物: WP10 · 标注
条件: licensed · 授权专业软件 long · 长流程
提案难度: 5 / 5

建议验证方式

Per-document coding vs gold: family-consistency violations = 0, responsiveness accuracy >= 0.95, privilege withhold flags exact; QC flag on 34 parents. Mass-edit and propagation via API disabled; History audit must show the layout.

建议部分得分

40% zero family conflicts; 35% responsiveness accuracy; 15% privilege flags; 10% QC flags.

R10.01.01.01-T07 Rate and issue-code ten hot documents in an Everlaw binder

原始指令

Everlaw sandbox: binder 'Deposition prep - J. Marlow (synthetic)' has 10 documents. Rate each Hot/Warm/Cold and apply issue codes from the project's 'Issues' category (Pricing, Notice, Damages, Spoliation) per input/issue_guide.pdf.

平台: web · 任意系统（网页）
应用: Everlaw
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 2 / 5

建议验证方式

Rating per document exact vs gold (accuracy >= 0.9); issue-code sets Jaccard >= 0.8 mean. Everlaw API tokens are withheld for the sandbox project, so changes must appear in the document history under the reviewer account.

建议部分得分

50% ratings; 50% issue-code Jaccard.

R10.01.01.01-T08 Assemble a chronology of key documents in Everlaw Storybuilder

原始指令

Everlaw sandbox: from the 'Hot' documents, build a Storybuilder story 'Surcharge timeline' with the 8 documents that establish notice and agreement dates, each as a dated event with a one-sentence description and the key quote highlighted in the document.

平台: web · 任意系统（网页）
应用: Everlaw
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单 path · 连续路径
工作产物: WP01 · 文档
条件: licensed · 授权专业软件
提案难度: 3 / 5

建议验证方式

Story export: event set vs gold 8 documents (F1 >= 0.85); event dates exact; each event has a highlight whose text overlaps the gold quote by >= 70% tokens; description quality scored by LLM rubric (15%). Everlaw API tokens are withheld, so events and highlights must appear in the project history under the reviewer account.

建议部分得分

40% correct documents; 25% dates; 20% highlights; 15% descriptions (LLM rubric).

R10.01.01.01-T09 Generate a privilege log from coded documents in DISCO

原始指令

DISCO sandbox matter 'Pinecrest HOA v. Valmont (synthetic)': 37 documents are tagged Privileged. Complete the privilege-log fields (date, author, recipients, privilege type, basis description) on each, exclude the 4 that a second reviewer downgraded to Not Privileged, and export the log to output/priv_log.xlsx in the format of input/log_template.xlsx.

平台: web · 任意系统（网页）
应用: DISCO · Excel
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: type · 文字与表单 click · 指向与点击 multi · 跨应用
工作产物: WP02 · 记录 / 表格
条件: licensed · 授权专业软件 multiapp · 多应用
提案难度: 4 / 5

建议验证方式

priv_log.xlsx: 33 rows (set equality on control number), date/author/recipients exact, privilege type exact; template columns preserved; basis text non-empty and does not reveal privileged content (keyword blacklist). DISCO API credentials are withheld, and the 33 entries must also exist as completed privilege-log fields in the DISCO matter, not only in the spreadsheet.

建议部分得分

30% correct row set; 40% metadata fields; 15% type; 15% template and safe descriptions.

R10.01.01.01-T10 Review only the inclusive emails of fifteen threads and propagate coding

原始指令

DISCO sandbox: saved search 'Threads-Q2' groups 15 email threads (96 messages). Review only the inclusive messages, code them Responsive/Not Responsive and Privileged per input/review_protocol.pdf, and propagate coding to non-inclusive duplicates in each thread as the protocol allows.

平台: web · 任意系统（网页）
应用: DISCO
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 nav · 平移 / 缩放 / 滚动
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 3 / 5

建议验证方式

Inclusive-message coding vs gold (F1 >= 0.85); non-inclusive messages carry propagated coding consistent with their inclusive (0 conflicts); messages outside the 15 threads untouched. DISCO API credentials are withheld from the reviewer, so coding must appear in the DISCO audit history under the reviewer's session.

建议部分得分

55% inclusive coding F1; 35% propagation consistency; 10% no stray edits.

R10.01.01.01-T11 Redact dates of birth inside spreadsheet natives

原始指令

Relativity sandbox: 6 Excel natives (payroll exports, synthetic) must be produced natively with DOB cells redacted. Use native spreadsheet redaction to redact every DOB cell (including a hidden column and a second sheet) with the text 'REDACTED-PII', leaving other cells intact.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 vis · 视觉判断 nav · 平移 / 缩放 / 滚动
工作产物: WP10 · 标注
条件: licensed · 授权专业软件 hidden · 隐藏 / 过期状态
提案难度: 4 / 5

建议验证方式

Native redaction markups vs gold cell ranges (recall 100%, precision >= 95%) including hidden column and second sheet; test production natives opened with openpyxl show no DOB values. REST API access is denied to the reviewer group, and redactions must exist as Relativity native markups in History, not as edited native files.

建议部分得分

50% visible DOB cells; 30% hidden column and second sheet; 20% no over-redaction.

R10.01.01.01-T12 Run a statistical QC sample over first-level coding and report the error rate

原始指令

Relativity sandbox: 2,000 documents were first-level coded. Create a random sample at 95% confidence / 5% margin with the sampling tool, re-review the sample, overturn incorrect responsiveness or privilege calls (field 'QC Overturn' with reason), and record the observed error rate and the projected count of mis-coded documents in output/qc_report.txt.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 nav · 平移 / 缩放 / 滚动 type · 文字与表单
工作产物: WP12 · 答案 / 决策
条件: licensed · 授权专业软件 long · 长流程
提案难度: 5 / 5

建议验证方式

Sample size equals the tool's value for the stated parameters; overturns vs gold errors within the sample (F1 >= 0.85); qc_report.txt error rate within 1 point of the rate implied by gold and projection consistent with it. REST API and Relativity Scripts access are denied to the reviewer group, so the sample and overturns must appear in History from the UI.

建议部分得分

15% sample created correctly; 55% overturn F1; 30% report numbers.

R10.01.01.01-T13 Code foreign-language documents using in-platform machine translation

原始指令

Everlaw sandbox: 15 German and Spanish documents in binder 'Foreign-language'. Use the platform's translation view to review them, code responsiveness and issue tags per input/issue_guide.pdf, and add a 'Needs certified translation' tag to the 3 most important responsive ones as defined in the guide.

平台: web · 任意系统（网页）
应用: Everlaw
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 nav · 平移 / 缩放 / 滚动
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 3 / 5

建议验证方式

Responsiveness F1 >= 0.85; issue-tag Jaccard >= 0.75; the 'certified translation' set equals the 3 gold documents. The agent gets no Everlaw API token, and coding history must show each code applied in the reviewer's browser session.

建议部分得分

45% responsiveness; 30% issue tags; 25% certified-translation set.

R10.01.01.01-T14 Flag documents for second-level review with reviewer notes

原始指令

Relativity sandbox, batch FL-011: tag the 8 documents that mention the custodian's personal medical leave as 'Needs Second Level' and add a reviewer note on each quoting the sentence that triggered the flag.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 1 / 5

建议验证方式

Tagged set equals the 8 gold documents (F1 >= 0.9); each note contains a sentence overlapping the gold trigger sentence by >= 70% tokens. REST API access and mass-edit permission are denied to the reviewer group, so History must show per-document tag and note edits.

建议部分得分

70% tagged set F1; 30% notes quote the trigger.

R10.01.01.01-T15 Redact handwritten account numbers on scanned checks

原始指令

Everlaw sandbox: 6 scanned check images (synthetic) must be produced with account and routing numbers redacted, including handwritten account numbers in memo lines, while payee, amount and date stay visible. Draw redactions with the 'PII - Financial' stamp.

平台: web · 任意系统（网页）
应用: Everlaw
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: geom · 精确几何 vis · 视觉判断 path · 连续路径
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 4 / 5

建议验证方式

Redaction boxes vs gold regions: 100% of gold number regions covered >= 95%; payee/amount/date regions overlap <= 5%; produced images run through OCR show no gold digit strings. Everlaw API tokens are withheld, and redactions must exist as Everlaw redaction annotations in the reviewer's history, not as altered image files.

建议部分得分

60% numbers covered; 25% keep-regions visible; 15% stamp label.

R10.01.01.01-T16 Handle a clawback: find copies of a produced privileged email and withhold them

原始指令

Relativity sandbox: opposing counsel returned DEF0004512 under the clawback order (input/clawback_letter.pdf). Find all exact and near duplicates (textual near-dup >= 90%) of that email in the workspace, code them Privileged/Withhold, add each to the 'Clawback Log' RDO with Bates or control number, and remove them from the open production set 'PROD-004'.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 nav · 平移 / 缩放 / 滚动 type · 文字与表单
工作产物: WP08 · 应用 / 系统状态
条件: licensed · 授权专业软件 risk · 高风险 / 不可逆
提案难度: 3 / 5

建议验证方式

Withheld set equals the 5 gold copies; Clawback Log rows exact; PROD-004 no longer contains them and its other 412 documents are unchanged; production not run. The agent has no Relativity REST/Kepler API credentials; coding, RDO and production-set changes must appear in the workspace audit log under its browser session.

建议部分得分

35% copies found; 20% coded privileged; 20% log; 25% removed from production set only.

R10.01.01.01-T17 Stage a redacted production set with branding and privilege slip sheets

原始指令

Relativity sandbox: create production set PROD-005 from saved search 'Ready to Produce' with Bates prefix KES, start number 0012001, footer branding (Bates left, 'CONFIDENTIAL' right), burned redactions, and slip sheets for withheld family members. Run the production checks and fix every conflict they report. Stop before the final production run and export.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 type · 文字与表单 nav · 平移 / 缩放 / 滚动
工作产物: WP08 · 应用 / 系统状态
条件: licensed · 授权专业软件 risk · 高风险 / 不可逆 long · 长流程
提案难度: 5 / 5

建议验证方式

Production set settings vs gold (prefix, start, branding, markup set, placeholders); document list equals gold after conflicts fixed (3 seeded: unredacted privileged, missing image, family split); state 'Staged', never 'Produced'. REST API and Relativity Scripts access are denied to the agent account, so the workspace audit must show the set configured from the UI.

建议部分得分

30% settings; 40% conflicts resolved and doc list exact; 30% stopped at staged state.

R10.01.01.01-T18 Train an Active Learning project to an elusion test

原始指令

Relativity sandbox: an Active Learning project 'Surcharge-AL' is set up on 8,000 documents. Review the prioritized queue (code Responsive/Not Responsive per protocol) until the project's rank stabilizes, then run an elusion test at 95% confidence and record the estimated recall and elusion rate in output/al_report.txt.

平台: web · 任意系统（网页）
应用: Relativity
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: click · 指向与点击 nav · 平移 / 缩放 / 滚动
工作产物: WP12 · 答案 / 决策
条件: licensed · 授权专业软件 long · 长流程
提案难度: 4 / 5

建议验证方式

Coding accuracy on reviewed documents >= 0.9 vs gold; an elusion test exists in the project; al_report.txt values equal the project's reported values; gold-computed true recall >= 0.75. REST API access and mass-edit are denied to the reviewer group, so History must show per-document coding from the Active Learning queue.

建议部分得分

40% coding accuracy; 30% elusion test run; 30% report values and recall threshold.

R10.01.01.01-T19 Code responsive segments in recorded calls by timestamp

原始指令

Everlaw sandbox: 4 recorded dispatcher calls (audio with machine transcripts, synthetic). Listen and, where the transcript is wrong, rely on the audio. Code each call Responsive if it discusses the surcharge, and add timestamped annotations marking start and end of each responsive segment.

平台: web · 任意系统（网页）
应用: Everlaw
交互表面: brow · 浏览器
GUI 需求 · 任务设定: G3 · 仅 GUI
GUI 需求 · 普通权限: G2 · GUI 必要
交互原语: av · 音视频时序 click · 指向与点击 type · 文字与表单
工作产物: WP10 · 标注
条件: licensed · 授权专业软件
提案难度: 3 / 5

建议验证方式

Call-level coding exact; segment annotations vs gold intervals with temporal IoU >= 0.6 each; one seeded mis-transcription means the segment is only findable from audio. Everlaw API tokens are withheld, so codes and timestamped annotations must appear in the activity history under the reviewer account.

建议部分得分

30% call-level coding; 60% segment IoU; 10% annotation labels.

R10.01.01.01-T20 Classify the same 50 emails as text files without a review platform (control)

原始指令

Control pair of T02. The same 50 emails and attachments are given as text in input/batch_FL007/ with input/review_protocol.pdf and input/privilege_names.csv. Write output/coding.csv with control number, responsive (Y/N), privileged (Y/N), privilege type and basis.

平台: lin · Linux
应用: Python
交互表面: term · 终端 api · API / SDK
GUI 需求 · 任务设定: AP · API 足够
GUI 需求 · 普通权限: CL · CLI 足够
交互原语: code · 代码与命令
工作产物: WP10 · 标注
条件: 未声明
提案难度: 3 / 5

建议验证方式

Same gold and thresholds as T02 (responsiveness and privilege F1). Isolates review judgment from platform operation.

建议部分得分

45% responsiveness F1; 40% privilege F1 and type; 15% basis present.

已有研究映射

以下沿用研究目录中的基准与小类映射，不是逐题标注、执行记录或产品背书。

BigLaw Bench

电子取证审阅、编码与涂黑

任务提案 · 20

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

原始指令

建议验证方式

建议部分得分

已有研究映射

同工作流族的小类