R10.01.01.01 · TASK CLASS

eDiscovery review, coding & redaction

Review document sets, apply issue codes and privilege redactions.

20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.

Typical applications: Relativity · Everlaw · DISCO
Class GUI-need label: G3 · GUI-only

Task proposals · 20

Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.

R10.01.01.01-T01 Code a 20-document batch for responsiveness in Relativity

Original instruction

Relativity sandbox workspace 'Halvorsen v. Kestrel Freight (synthetic)'. Check out review batch FL-001 (20 emails) and code each Responsive or Not Responsive on the 'First Level' coding layout following input/review_protocol.pdf section 2 (responsive = discusses the 2024 fuel-surcharge agreement). Check the batch back in.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 2 / 5

Proposed verifier

Responsiveness field per document vs gold (accuracy >= 0.9); batch status checked in. REST API access and mass-edit permission are denied to the reviewer group, so Relativity History must show per-document edits from the coding layout.

Proposed partial credit

80% coding accuracy (scaled from 0.6 to 1.0); 20% batch checked in.

R10.01.01.01-T02 Review a 50-email batch for responsiveness and privilege

Original instruction

Relativity sandbox, batch FL-007 (50 emails with attachments). Per input/review_protocol.pdf code each document Responsive/Not Responsive and, where applicable, Privileged with type (Attorney-Client, Work Product) and a one-line privilege basis. Outside counsel names are listed in input/privilege_names.csv.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms nav · Pan/zoom/scroll
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 3 / 5

Proposed verifier

Responsiveness F1 >= 0.85 and privilege F1 >= 0.85 vs gold; privilege type exact on true positives; basis text non-empty. Coding must come from the layout (History audit; no mass edit, no API).

Proposed partial credit

45% responsiveness F1; 40% privilege F1 and type; 15% privilege basis present.

R10.01.01.01-T03 Redact Social Security numbers across ten imaged documents

Original instruction

Relativity sandbox: saved search 'PII-SSN-Imaged' returns 10 imaged documents (scanned HR forms and letters, synthetic). Using the image viewer, draw black-box redactions with the label 'PII' over every SSN, including handwritten ones, without covering adjacent names or dates.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: geom · Precise geometry vis · Visual judgment click · Point & click
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 3 / 5

Proposed verifier

Markup table: redaction boxes vs gold SSN boxes, each gold box covered with IoU >= 0.5 and >= 95% coverage; OCR of a test production shows no 9-digit SSN pattern; boxes covering gold 'keep' regions (names/dates) count as over-redaction (max 1). REST API access is denied to the reviewer group and automated (search-term) redaction is disabled, so History must show markups drawn in the image viewer.

Proposed partial credit

60% all SSNs covered; 20% labels correct; 20% no over-redaction.

R10.01.01.01-T04 Apply partial privilege redactions and log them

Original instruction

Everlaw sandbox project: 12 responsive emails contain a paragraph quoting outside counsel's advice. Redact only those paragraphs with the 'Privilege - AC' stamp, code each document 'Produce with redactions', and fill the redaction-log fields (date, author, recipients, privilege basis) for each.

Platforms: web · Any OS (web)
Applications: Everlaw
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: geom · Precise geometry vis · Visual judgment type · Text & forms
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 4 / 5

Proposed verifier

Redaction export: per document, redacted region covers the gold privileged paragraph (coverage >= 95%) and leaves >= 90% of the non-privileged text visible; coding exact; redaction-log fields vs gold (dates/authors exact). Project API is disabled for the reviewer.

Proposed partial credit

50% redaction precision/recall; 20% coding; 30% log fields.

R10.01.01.01-T05 Build a persistent highlight set and a saved search for review batches

Original instruction

Relativity sandbox: create a persistent highlight set 'Surcharge Terms' with the 6 terms in input/terms.txt (two with proximity: fuel w/5 surcharge, rebate w/3 tier) in distinct colors, then a saved search returning unreviewed documents hitting any term, and a batch set of 25-document batches from it.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP08 · App/system state
Conditions: licensed · Licensed pro app
Proposed difficulty: 3 / 5

Proposed verifier

Workspace objects: highlight set with 6 terms and syntax exact; saved search conditions return the gold document set (set equality); batch set with size 25 and batch count = ceil(hits/25). REST API and Relativity Scripts access are denied to the agent account, so the workspace audit must show each object created from the UI.

Proposed partial credit

35% highlight set; 40% saved search returns gold set; 25% batch set.

R10.01.01.01-T06 Code 120 documents with family consistency and resolve reviewer conflicts

Original instruction

Relativity sandbox, batch set 'Families-QC': 120 documents in 34 families, partially coded by two reviewers with conflicts. Apply input/review_protocol.pdf family rules (a family is produced if any member is responsive; privileged members withheld with slip sheet), recode conflicting members, and set 'Family QC Complete' on each family parent.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll vis · Visual judgment
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Per-document coding vs gold: family-consistency violations = 0, responsiveness accuracy >= 0.95, privilege withhold flags exact; QC flag on 34 parents. Mass-edit and propagation via API disabled; History audit must show the layout.

Proposed partial credit

40% zero family conflicts; 35% responsiveness accuracy; 15% privilege flags; 10% QC flags.

R10.01.01.01-T07 Rate and issue-code ten hot documents in an Everlaw binder

Original instruction

Everlaw sandbox: binder 'Deposition prep - J. Marlow (synthetic)' has 10 documents. Rate each Hot/Warm/Cold and apply issue codes from the project's 'Issues' category (Pricing, Notice, Damages, Spoliation) per input/issue_guide.pdf.

Platforms: web · Any OS (web)
Applications: Everlaw
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 2 / 5

Proposed verifier

Rating per document exact vs gold (accuracy >= 0.9); issue-code sets Jaccard >= 0.8 mean. Everlaw API tokens are withheld for the sandbox project, so changes must appear in the document history under the reviewer account.

Proposed partial credit

50% ratings; 50% issue-code Jaccard.

R10.01.01.01-T08 Assemble a chronology of key documents in Everlaw Storybuilder

Original instruction

Everlaw sandbox: from the 'Hot' documents, build a Storybuilder story 'Surcharge timeline' with the 8 documents that establish notice and agreement dates, each as a dated event with a one-sentence description and the key quote highlighted in the document.

Platforms: web · Any OS (web)
Applications: Everlaw
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms path · Continuous path
Work product: WP01 · Document
Conditions: licensed · Licensed pro app
Proposed difficulty: 3 / 5

Proposed verifier

Story export: event set vs gold 8 documents (F1 >= 0.85); event dates exact; each event has a highlight whose text overlaps the gold quote by >= 70% tokens; description quality scored by LLM rubric (15%). Everlaw API tokens are withheld, so events and highlights must appear in the project history under the reviewer account.

Proposed partial credit

40% correct documents; 25% dates; 20% highlights; 15% descriptions (LLM rubric).

R10.01.01.01-T09 Generate a privilege log from coded documents in DISCO

Original instruction

DISCO sandbox matter 'Pinecrest HOA v. Valmont (synthetic)': 37 documents are tagged Privileged. Complete the privilege-log fields (date, author, recipients, privilege type, basis description) on each, exclude the 4 that a second reviewer downgraded to Not Privileged, and export the log to output/priv_log.xlsx in the format of input/log_template.xlsx.

Platforms: web · Any OS (web)
Applications: DISCO · Excel
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: type · Text & forms click · Point & click multi · Cross-app
Work product: WP02 · Records/table
Conditions: licensed · Licensed pro app multiapp · Multi-app
Proposed difficulty: 4 / 5

Proposed verifier

priv_log.xlsx: 33 rows (set equality on control number), date/author/recipients exact, privilege type exact; template columns preserved; basis text non-empty and does not reveal privileged content (keyword blacklist). DISCO API credentials are withheld, and the 33 entries must also exist as completed privilege-log fields in the DISCO matter, not only in the spreadsheet.

Proposed partial credit

30% correct row set; 40% metadata fields; 15% type; 15% template and safe descriptions.

R10.01.01.01-T10 Review only the inclusive emails of fifteen threads and propagate coding

Original instruction

DISCO sandbox: saved search 'Threads-Q2' groups 15 email threads (96 messages). Review only the inclusive messages, code them Responsive/Not Responsive and Privileged per input/review_protocol.pdf, and propagate coding to non-inclusive duplicates in each thread as the protocol allows.

Platforms: web · Any OS (web)
Applications: DISCO
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 3 / 5

Proposed verifier

Inclusive-message coding vs gold (F1 >= 0.85); non-inclusive messages carry propagated coding consistent with their inclusive (0 conflicts); messages outside the 15 threads untouched. DISCO API credentials are withheld from the reviewer, so coding must appear in the DISCO audit history under the reviewer's session.

Proposed partial credit

55% inclusive coding F1; 35% propagation consistency; 10% no stray edits.

R10.01.01.01-T11 Redact dates of birth inside spreadsheet natives

Original instruction

Relativity sandbox: 6 Excel natives (payroll exports, synthetic) must be produced natively with DOB cells redacted. Use native spreadsheet redaction to redact every DOB cell (including a hidden column and a second sheet) with the text 'REDACTED-PII', leaving other cells intact.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click vis · Visual judgment nav · Pan/zoom/scroll
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app hidden · Hidden/stale state
Proposed difficulty: 4 / 5

Proposed verifier

Native redaction markups vs gold cell ranges (recall 100%, precision >= 95%) including hidden column and second sheet; test production natives opened with openpyxl show no DOB values. REST API access is denied to the reviewer group, and redactions must exist as Relativity native markups in History, not as edited native files.

Proposed partial credit

50% visible DOB cells; 30% hidden column and second sheet; 20% no over-redaction.

R10.01.01.01-T12 Run a statistical QC sample over first-level coding and report the error rate

Original instruction

Relativity sandbox: 2,000 documents were first-level coded. Create a random sample at 95% confidence / 5% margin with the sampling tool, re-review the sample, overturn incorrect responsiveness or privilege calls (field 'QC Overturn' with reason), and record the observed error rate and the projected count of mis-coded documents in output/qc_report.txt.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll type · Text & forms
Work product: WP12 · Answer/decision
Conditions: licensed · Licensed pro app long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Sample size equals the tool's value for the stated parameters; overturns vs gold errors within the sample (F1 >= 0.85); qc_report.txt error rate within 1 point of the rate implied by gold and projection consistent with it. REST API and Relativity Scripts access are denied to the reviewer group, so the sample and overturns must appear in History from the UI.

Proposed partial credit

15% sample created correctly; 55% overturn F1; 30% report numbers.

R10.01.01.01-T13 Code foreign-language documents using in-platform machine translation

Original instruction

Everlaw sandbox: 15 German and Spanish documents in binder 'Foreign-language'. Use the platform's translation view to review them, code responsiveness and issue tags per input/issue_guide.pdf, and add a 'Needs certified translation' tag to the 3 most important responsive ones as defined in the guide.

Platforms: web · Any OS (web)
Applications: Everlaw
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 3 / 5

Proposed verifier

Responsiveness F1 >= 0.85; issue-tag Jaccard >= 0.75; the 'certified translation' set equals the 3 gold documents. The agent gets no Everlaw API token, and coding history must show each code applied in the reviewer's browser session.

Proposed partial credit

45% responsiveness; 30% issue tags; 25% certified-translation set.

R10.01.01.01-T14 Flag documents for second-level review with reviewer notes

Original instruction

Relativity sandbox, batch FL-011: tag the 8 documents that mention the custodian's personal medical leave as 'Needs Second Level' and add a reviewer note on each quoting the sentence that triggered the flag.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 1 / 5

Proposed verifier

Tagged set equals the 8 gold documents (F1 >= 0.9); each note contains a sentence overlapping the gold trigger sentence by >= 70% tokens. REST API access and mass-edit permission are denied to the reviewer group, so History must show per-document tag and note edits.

Proposed partial credit

70% tagged set F1; 30% notes quote the trigger.

R10.01.01.01-T15 Redact handwritten account numbers on scanned checks

Original instruction

Everlaw sandbox: 6 scanned check images (synthetic) must be produced with account and routing numbers redacted, including handwritten account numbers in memo lines, while payee, amount and date stay visible. Draw redactions with the 'PII - Financial' stamp.

Platforms: web · Any OS (web)
Applications: Everlaw
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: geom · Precise geometry vis · Visual judgment path · Continuous path
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 4 / 5

Proposed verifier

Redaction boxes vs gold regions: 100% of gold number regions covered >= 95%; payee/amount/date regions overlap <= 5%; produced images run through OCR show no gold digit strings. Everlaw API tokens are withheld, and redactions must exist as Everlaw redaction annotations in the reviewer's history, not as altered image files.

Proposed partial credit

60% numbers covered; 25% keep-regions visible; 15% stamp label.

R10.01.01.01-T16 Handle a clawback: find copies of a produced privileged email and withhold them

Original instruction

Relativity sandbox: opposing counsel returned DEF0004512 under the clawback order (input/clawback_letter.pdf). Find all exact and near duplicates (textual near-dup >= 90%) of that email in the workspace, code them Privileged/Withhold, add each to the 'Clawback Log' RDO with Bates or control number, and remove them from the open production set 'PROD-004'.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll type · Text & forms
Work product: WP08 · App/system state
Conditions: licensed · Licensed pro app risk · Irreversible/risky
Proposed difficulty: 3 / 5

Proposed verifier

Withheld set equals the 5 gold copies; Clawback Log rows exact; PROD-004 no longer contains them and its other 412 documents are unchanged; production not run. The agent has no Relativity REST/Kepler API credentials; coding, RDO and production-set changes must appear in the workspace audit log under its browser session.

Proposed partial credit

35% copies found; 20% coded privileged; 20% log; 25% removed from production set only.

R10.01.01.01-T17 Stage a redacted production set with branding and privilege slip sheets

Original instruction

Relativity sandbox: create production set PROD-005 from saved search 'Ready to Produce' with Bates prefix KES, start number 0012001, footer branding (Bates left, 'CONFIDENTIAL' right), burned redactions, and slip sheets for withheld family members. Run the production checks and fix every conflict they report. Stop before the final production run and export.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click type · Text & forms nav · Pan/zoom/scroll
Work product: WP08 · App/system state
Conditions: licensed · Licensed pro app risk · Irreversible/risky long · Long-horizon
Proposed difficulty: 5 / 5

Proposed verifier

Production set settings vs gold (prefix, start, branding, markup set, placeholders); document list equals gold after conflicts fixed (3 seeded: unredacted privileged, missing image, family split); state 'Staged', never 'Produced'. REST API and Relativity Scripts access are denied to the agent account, so the workspace audit must show the set configured from the UI.

Proposed partial credit

30% settings; 40% conflicts resolved and doc list exact; 30% stopped at staged state.

R10.01.01.01-T18 Train an Active Learning project to an elusion test

Original instruction

Relativity sandbox: an Active Learning project 'Surcharge-AL' is set up on 8,000 documents. Review the prioritized queue (code Responsive/Not Responsive per protocol) until the project's rank stabilizes, then run an elusion test at 95% confidence and record the estimated recall and elusion rate in output/al_report.txt.

Platforms: web · Any OS (web)
Applications: Relativity
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: click · Point & click nav · Pan/zoom/scroll
Work product: WP12 · Answer/decision
Conditions: licensed · Licensed pro app long · Long-horizon
Proposed difficulty: 4 / 5

Proposed verifier

Coding accuracy on reviewed documents >= 0.9 vs gold; an elusion test exists in the project; al_report.txt values equal the project's reported values; gold-computed true recall >= 0.75. REST API access and mass-edit are denied to the reviewer group, so History must show per-document coding from the Active Learning queue.

Proposed partial credit

40% coding accuracy; 30% elusion test run; 30% report values and recall threshold.

R10.01.01.01-T19 Code responsive segments in recorded calls by timestamp

Original instruction

Everlaw sandbox: 4 recorded dispatcher calls (audio with machine transcripts, synthetic). Listen and, where the transcript is wrong, rely on the audio. Code each call Responsive if it discusses the surcharge, and add timestamped annotations marking start and end of each responsive segment.

Platforms: web · Any OS (web)
Applications: Everlaw
Interfaces: brow · Browser
GUI need · task setting: G3 · GUI-only
GUI need · ordinary access: G2 · GUI-essential
Interaction primitives: av · Audio/video over time click · Point & click type · Text & forms
Work product: WP10 · Annotations
Conditions: licensed · Licensed pro app
Proposed difficulty: 3 / 5

Proposed verifier

Call-level coding exact; segment annotations vs gold intervals with temporal IoU >= 0.6 each; one seeded mis-transcription means the segment is only findable from audio. Everlaw API tokens are withheld, so codes and timestamped annotations must appear in the activity history under the reviewer account.

Proposed partial credit

30% call-level coding; 60% segment IoU; 10% annotation labels.

R10.01.01.01-T20 Classify the same 50 emails as text files without a review platform (control)

Original instruction

Control pair of T02. The same 50 emails and attachments are given as text in input/batch_FL007/ with input/review_protocol.pdf and input/privilege_names.csv. Write output/coding.csv with control number, responsive (Y/N), privileged (Y/N), privilege type and basis.

Platforms: lin · Linux
Applications: Python
Interfaces: term · Terminal api · API/SDK
GUI need · task setting: AP · API-sufficient
GUI need · ordinary access: CL · CLI-sufficient
Interaction primitives: code · Code & commands
Work product: WP10 · Annotations
Conditions: None declared
Proposed difficulty: 3 / 5

Proposed verifier

Same gold and thresholds as T02 (responsiveness and privilege F1). Isolates review judgment from platform operation.

Proposed partial credit

45% responsiveness F1; 40% privilege F1 and type; 15% basis present.

Mapped research references

Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.

BigLaw Bench

eDiscovery review, coding & redaction

Task proposals · 20

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Original instruction

Proposed verifier

Proposed partial credit

Mapped research references

In this workflow family