Case-law & statute research with citations
Find controlling authority and verify every citation.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Westlaw · Lexis+ · CourtListener
- Class GUI-need label
HY· Hybrid
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R10.02.01.01-T01 Retrieve a current regulation's operative text and last-amended citation
Original instruction
A client asks how many hours of service an employee needs for FMLA eligibility. Using eCFR in the browser, find the operative provision in 29 C.F.R. Part 825, and record in output/answer.json: the section and paragraph citation, the hour threshold, the quoted operative sentence, and the Federal Register citation listed as the section's most recent amendment source.
- Platforms
web· Any OS (web)- Applications
- eCFR · govinfo
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
answer.json fields vs curated key: paragraph cite exact, number exact, quote string-match >= 0.95 to the eCFR text as of the frozen snapshot date, FR citation exact. Snapshot pinned so answers do not drift.
Proposed partial credit
35% citation; 25% threshold; 20% quote; 20% FR source.
R10.02.01.01-T02 Find the controlling appellate decision on an issue and give its pin cite
Original instruction
Issue memo input/issue_delegation.pdf asks whether, in the Ninth Circuit, a court or the arbitrator decides arbitrability when the clause incorporates AAA rules. Using Westlaw (or Lexis+), identify the controlling published Ninth Circuit decision, its reporter citation, the pin cite for the holding, and whether it remains good law. Write output/authority.json.
- Platforms
web· Any OS (web)- Applications
- Westlaw · Lexis+
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsclick· Point & click- Work product
WP12· Answer/decision- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Curated key: case name and reporter citation exact; pin cite within the accepted page set; good-law status matches the citator flag on the snapshot date. Fabricated or non-precedential cites score 0 for that field.
Proposed partial credit
45% correct case; 25% reporter cite; 20% pin cite; 10% status.
R10.02.01.01-T03 Run citator checks on twelve authorities and flag negative treatment
Original instruction
input/brief_draft.docx cites 12 cases. Run KeyCite on each in Westlaw. For every case with red or yellow flag, record the flag, the most serious negative citing case (name and citation) and whether the negative treatment concerns the proposition the brief cites it for. Save output/citator_table.xlsx.
- Platforms
web· Any OS (web)- Applications
- Westlaw · Excel
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollmulti· Cross-app- Work product
WP02· Records/table- Conditions
licensed· Licensed pro appmultiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Rows vs curated key: flag color per case exact (12); negative citing case exact for the 5 flagged cases; proposition-relevance Y/N exact. Westlaw API access is denied to the account, and the research history export must confirm a KeyCite view for each of the 12 cases.
Proposed partial credit
35% flags; 40% negative citing cases; 25% relevance calls.
R10.02.01.01-T04 Cite-check a 30-citation brief including fabricated authorities
Original instruction
input/opposition_brief.docx contains 30 citations to cases and statutes. Verify each: the authority exists, volume/reporter/page are right, the pin cite supports the quoted or paraphrased text, and Bluebook form. Several citations may be fabricated. Produce output/citecheck.xlsx (cite, exists Y/N, corrected cite, pin OK Y/N, issue) using CourtListener plus Westlaw or Lexis+.
- Platforms
web· Any OS (web)- Applications
- CourtListener · Westlaw · Lexis+
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsmulti· Cross-app- Work product
WP02· Records/table- Conditions
licensed· Licensed pro appmultiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Per-row vs curated key: existence exact (5 seeded fabrications must be N), corrected cite exact for 8 seeded mis-cites, pin-support flag exact for 6 seeded misquotes; F1 over seeded issues >= 0.85.
Proposed partial credit
35% fabrications caught; 30% corrected cites; 25% pin/quote problems; 10% Bluebook fixes.
R10.02.01.01-T05 Trace an amendment to its public law and Statutes at Large citation
Original instruction
The current text of 15 U.S.C. section 1681c(a)(5) was changed by a later act. Using Congress.gov, govinfo and the U.S. Code notes, record in output/history.json the amending public law number, its Statutes at Large citation, enactment date, and the exact words added or removed.
- Platforms
web· Any OS (web)- Applications
- Congress.gov · govinfo · uscode.house.gov
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Curated key: Pub. L. number, Stat. cite and date exact; changed text token-F1 >= 0.9 against the enrolled bill language. Snapshot of sites pinned.
Proposed partial credit
30% public law; 25% Stat. cite; 15% date; 30% changed text.
R10.02.01.01-T06 Build a fifteen-state survey of contract limitation periods with verified citations
Original instruction
For the 15 states in input/states.txt, find the statute of limitations for actions on written contracts (and any separate period for sale-of-goods contracts under UCC 2-725 as adopted). Record period, code citation, and one appellate case applying it with reporter cite, in output/sol_survey.xlsx. Use Lexis+ or Westlaw and verify every code section on the official legislature site.
- Platforms
web· Any OS (web)- Applications
- Lexis+ · Westlaw · state legislature sites · Excel
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsmulti· Cross-app- Work product
WP02· Records/table- Conditions
licensed· Licensed pro applong· Long-horizonmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Curated key per state: period exact; code citation exact; UCC period and cite exact; case cite must exist (grader checks via CourtListener citation lookup) and apply the cited section (curated acceptable set). Pass >= 90% of cells.
Proposed partial credit
35% periods; 30% code cites; 15% UCC rows; 20% valid applying cases.
R10.02.01.01-T07 Pull an opinion and its docket from CourtListener
Original instruction
In CourtListener, find the opinion in docket 'Halvorsen v. Kestrel Freight' (seeded mirror case) decided in 2025, download the opinion PDF to output/opinion.pdf, and record in output/meta.json the court, date filed, author judge, citation if published, and the docket number.
- Platforms
web· Any OS (web)- Applications
- CourtListener
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 1 / 5
Proposed verifier
opinion.pdf hash equals the mirror file; meta.json fields exact against the mirror's database row.
Proposed partial credit
40% correct PDF; 60% metadata fields (pro rata per field).
R10.02.01.01-T08 Find the five most on-point circuit decisions interpreting a statutory element
Original instruction
Research question input/q_ada_qualified.txt: how the Seventh Circuit has applied the 'essential functions' element of 42 U.S.C. 12111(8) to attendance requirements since 2018. Using Westlaw or Lexis+ (headnotes, citing references), list the 5 most on-point published decisions with citation, pin cite and a one-sentence holding in output/top5.json.
- Platforms
web· Any OS (web)- Applications
- Westlaw · Lexis+
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsclick· Point & click- Work product
WP12· Answer/decision- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Precision of the 5 cases against an expert-curated relevant set (>= 4 of 5 in set to pass); citations exact; pin cites within the accepted page ranges; holding sentences scored by LLM rubric (only 15%).
Proposed partial credit
55% precision vs curated set; 20% citations; 10% pin cites; 15% holdings (LLM rubric).
R10.02.01.01-T09 Compare two point-in-time versions of a regulation and source the change
Original instruction
Using eCFR's point-in-time and compare views, determine how 40 C.F.R. section 60.5397a changed between 2023-01-01 and the snapshot date. Write output/reg_change.json with each changed paragraph, old and new text, the Federal Register citation and effective date of each amending rule, and produce a redline PDF output/redline.pdf from the site's compare view.
- Platforms
web· Any OS (web)- Applications
- eCFR · Federal Register
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Changed-paragraph set exact vs key; old/new text token-F1 >= 0.95; FR cites and effective dates exact; redline.pdf exists and contains the insertion/deletion markup for those paragraphs.
Proposed partial credit
35% changed paragraphs; 25% texts; 25% FR sources; 15% redline export.
R10.02.01.01-T10 Organize a Westlaw research folder with highlighted holdings
Original instruction
In Westlaw, create folder 'Kestrel - Fuel Surcharge' with subfolders 'Contract formation' and 'Damages'. Save the 8 authorities listed in input/authorities.txt into the right subfolders, highlight the holding passage in each and add a note naming the proposition, then export the folder's annotations report to output/folder_report.pdf.
- Platforms
web· Any OS (web)- Applications
- Westlaw
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickpath· Continuous pathtype· Text & forms- Work product
WP10· Annotations- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
folder_report.pdf parsed: folder structure and 8 documents in correct subfolders; each has a highlight overlapping the curated holding passage (>= 60% token overlap) and a note. Westlaw API access is denied to the account, so folders, highlights and notes can be created only in the Westlaw UI; the report must be Westlaw's own export.
Proposed partial credit
30% folders and placement; 45% highlights on holdings; 25% notes and report.
R10.02.01.01-T11 Supply star-paginated pin cites for ten quotations
Original instruction
input/memo_quotes.docx has 10 quotations from 6 Supreme Court opinions with only 'U.S. ___' placeholders for pin cites. Using star pagination in Westlaw or Lexis+, insert the exact U.S. Reports page (and S. Ct. parallel page) where each quoted passage begins, and save output/memo_pinned.docx.
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- Westlaw · Lexis+ · Word
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentmulti· Cross-app- Work product
WP01· Document- Conditions
licensed· Licensed pro appmultiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Each of 10 pin cites equals the curated U.S. page (passages spanning a page break accept the starting page) and S. Ct. page; no other text changed in the docx. Westlaw and Lexis+ API access is denied, so star pages are reachable only in the research UI; research history must show the 6 opinions viewed in the session.
Proposed partial credit
60% U.S. pins; 30% S. Ct. pins; 10% document otherwise unchanged.
R10.02.01.01-T12 Write a choice-of-law research memo with fully verified authorities
Original instruction
input/assignment.pdf: a Delaware-law choice-of-law clause in a contract performed in California; does California's non-compete ban (Bus. & Prof. Code 16600, 16600.5) override it? Research both states, then write output/memo.docx (question, short answer, analysis, table of authorities) citing only authorities you verified, each with pin cites.
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- Westlaw · Lexis+ · Word
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsmulti· Cross-app- Work product
WP01· Document- Conditions
licensed· Licensed pro applong· Long-horizonmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Every citation in memo.docx must resolve (grader citation lookup) and appear in the TOA; recall of 6 must-cite authorities; short answer matches the curated conclusion; analysis quality by LLM rubric (25%). Any fabricated cite caps the score at 30%.
Proposed partial credit
30% must-cite recall; 25% no fabricated/mis-cited authority; 20% short answer; 25% analysis (LLM rubric).
R10.02.01.01-T13 Determine whether a state statute was amended or repealed and cite the session law
Original instruction
A 2019 memo relies on Texas Property Code section 92.0563. Using the Texas Legislature Online site and the current code, determine whether the section was amended or repealed after 2019, identify the session law (bill number, chapter, session) and effective date, and record the current text in output/currency.json.
- Platforms
web· Any OS (web)- Applications
- Texas Legislature Online · statutes.capitol.texas.gov
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Curated key: amended/repealed status exact; bill, chapter, session and effective date exact; current text token-F1 >= 0.95 against the snapshot.
Proposed partial credit
30% status; 45% session-law details; 25% current text.
R10.02.01.01-T14 Convert fifteen citations to Bluebook form with parallel cites
Original instruction
input/raw_cites.txt lists 15 citations in mixed styles (neutral cites, Westlaw cites, state reporters). Using CourtListener's citation lookup and the court's rules for parallel cites in input/local_rule_cites.pdf, rewrite each in Bluebook form with the required parallel citation and court/year parenthetical. Save output/cites_bluebook.txt.
- Platforms
web· Any OS (web)- Applications
- CourtListener
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & forms- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Each line string-compared to the curated Bluebook form after whitespace normalization; partial credit per element (reporter, page, parallel, parenthetical).
Proposed partial credit
40% primary reporter cites; 30% parallel cites; 30% parentheticals.
R10.02.01.01-T15 Locate an unpublished district order in RECAP and cite it properly
Original instruction
A colleague remembers a 2024 order in 'Pinecrest HOA v. Valmont' (seeded mirror case, D. Colo.) denying class certification. Find the docket in CourtListener RECAP, identify the docket entry number and date of the order, download it to output/order.pdf, and write its proper citation (no reporter: docket number, ECF No., court, date) in output/cite.txt.
- Platforms
web· Any OS (web)- Applications
- CourtListener RECAP
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
order.pdf hash equals the mirror document; cite.txt matches the curated citation (case name, No., ECF No., court, exact date) after normalization. A similarly named order granting a different motion is seeded as a distractor.
Proposed partial credit
50% correct document; 50% citation elements.
R10.02.01.01-T16 Use a practice guide to find leading cases and confirm they are good law
Original instruction
In Lexis+ open the practice-guide/treatise section on 'implied covenant of good faith - termination at will' for New York. Record the section citation, the 3 leading New York Court of Appeals cases it relies on, and run Shepard's on each to confirm treatment. Save output/treatise_cases.json.
- Platforms
web· Any OS (web)- Applications
- Lexis+
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & click- Work product
WP12· Answer/decision- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Treatise section cite exact; 3 cases equal the curated set (order-free) with citations exact; Shepard's signal per case matches the snapshot. Lexis+ API is not available to the account.
Proposed partial credit
25% section cite; 50% cases; 25% Shepard's signals.
R10.02.01.01-T17 Audit an AI-drafted brief for hallucinated authorities and correct it with tracked changes
Original instruction
input/ai_draft_brief.docx (synthetic matter) has 40 citations; some cases do not exist, some are real but misquoted, and some have wrong pin cites or holdings. Verify each with CourtListener and Westlaw/Lexis+, then fix the brief in Word with Track Changes: replace or delete bad authorities, correct pins and quotes, and add a comment on each change explaining it.
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- CourtListener · Westlaw · Word
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsmulti· Cross-app- Work product
WP01· Document- Conditions
licensed· Licensed pro applong· Long-horizonmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
docx revision XML: every seeded problem (7 fabricated, 6 misquoted, 5 wrong pin) has a w:ins/w:del change at that citation and an anchored comment; replacement authorities resolve and support the proposition (curated acceptable set); no new unverifiable cites.
Proposed partial credit
35% fabricated cites removed/replaced; 25% misquotes fixed; 20% pins fixed; 20% comments and no new errors.
R10.02.01.01-T18 Trace a decision's subsequent history and write the full citation
Original instruction
For the 6 circuit decisions in input/cases_history.txt, use KeyCite or Shepard's history views to find all subsequent history (rehearing en banc, reversed, cert. denied with date) and write each full Bluebook citation with subsequent history in output/history_cites.txt.
- Platforms
web· Any OS (web)- Applications
- Westlaw · Lexis+
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & clicktype· Text & forms- Work product
WP01· Document- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
Per case vs curated: each history element (action, court, citation, year) exact; cert-denied citations include the S. Ct. cite where available on the snapshot date; Bluebook ordering correct. No Westlaw/Lexis API key is issued; every history element must appear in the agent's KeyCite/Shepard's views in the UI session log, so cites taken from other sources fail.
Proposed partial credit
60% history elements; 25% citations; 15% formatting.
R10.02.01.01-T19 Verify the same twelve citations through the CourtListener API (control)
Original instruction
Control pair of T03. Using the CourtListener REST API (token in env CL_TOKEN), resolve each of the 12 citations from input/brief_cites.txt, list citing opinions, and classify any negative treatment you can establish from the citing opinions' text; write output/citator_api.csv.
- Platforms
lin· Linux- Applications
- CourtListener API · Python
- Interfaces
api· API/SDKterm· Terminal- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
code· Code & commands- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Same key as T03 for existence and negative citing case; flags derived from citing text. Measures what a public API recovers versus the commercial citator GUI.
Proposed partial credit
30% citations resolved; 45% negative citing cases; 25% classification.
R10.02.01.01-T20 Find interpreting cases through statutory annotations and verify each
Original instruction
In Westlaw's annotated U.S. Code, open the Notes of Decisions for 47 U.S.C. section 227(a)(1) and identify the 3 most recent published circuit decisions interpreting 'automatic telephone dialing system' after the Supreme Court's 2021 decision on the term. Record citation, pin cite and holding sentence, confirm each is good law, in output/annotations.json.
- Platforms
web· Any OS (web)- Applications
- Westlaw
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & click- Work product
WP12· Answer/decision- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
3 cases equal the curated set as of the snapshot (order-free); citations and pin cites exact; KeyCite status matches snapshot; holding sentence overlaps curated holding by >= 60% tokens. Westlaw API access is denied to the account, and research history must show the Notes of Decisions view and a KeyCite view for each case from the session.
Proposed partial credit
50% correct cases; 25% cites and pins; 10% status; 15% holding sentences.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.