Contract redlining & deal points
Compare and redline drafts and extract deal points.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Word (Compare) · Litera Compare
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R10.02.01.02-T01 Produce a Word Compare redline of two NDA versions
Original instruction
input/nda_v1.docx (our draft) and input/nda_v2.docx (counterparty return, untracked). In Word, run Compare with v1 as original and v2 as revised, label changes by author 'Counterparty', show formatting changes off, and save the comparison document as output/nda_redline_v1_v2.docx.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & click- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Revision XML: the set of w:ins/w:del text spans equals the gold diff (token F1 >= 0.98); revision author 'Counterparty'; no formatting revisions (w:rPrChange) present; docProps/app.xml from Word with rsids (python-docx output lacks them). python-docx and other OOXML libraries are not installed, and Word macros/VBA and external automation (COM/AppleScript) are disabled for the agent.
Proposed partial credit
70% revision set; 15% author label; 15% formatting changes excluded.
R10.02.01.02-T02 Accept every counterparty change except the governing-law change
Original instruction
input/nda_redline.docx contains 14 tracked changes from the counterparty. Accept all of them except the change to the governing-law clause (Section 11), which you reject, and add a comment on Section 11 reading 'Governing law must remain Delaware.' Save as output/nda_v3.docx with no remaining revisions.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrolltype· Text & forms- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Resulting text equals gold v3 (13 accepted, 1 rejected); zero w:ins/w:del left; one comment anchored within Section 11 with exact text. VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (app metadata and rsids present).
Proposed partial credit
60% correct accept/reject result; 20% no remaining revisions; 20% comment anchored.
R10.02.01.02-T03 Redline an indemnity clause to add a twelve-month fee cap
Original instruction
input/msa_draft.docx Section 9 (Indemnification) has no liability cap. With Track Changes on as author 'Buyer Counsel', insert a cap limiting aggregate indemnity liability to fees paid in the 12 months preceding the claim, carve out fraud and confidentiality breaches, and add a comment explaining the change. Do not alter other sections.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & click- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Revisions only inside Section 9 by 'Buyer Counsel'; inserted text contains the 12-month fee cap and both carve-outs (pattern checks); comment anchored in Section 9; text outside Section 9 byte-equal to input. Drafting quality checked by LLM rubric (15%). VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (rsids present).
Proposed partial credit
45% cap and carve-outs tracked; 20% scope limited to Section 9; 20% comment; 15% drafting quality (LLM rubric).
R10.02.01.02-T04 Respond to a counterparty markup per an issues list
Original instruction
input/supply_agmt_cp_markup.docx has 11 tracked changes from the supplier. input/issues_list.xlsx says which to accept (6), reject (3) and counter (2, with our counter-language). Process each accordingly, layering our counter-proposals as new tracked changes by 'Customer Counsel' with a comment, and leave the document with only our open revisions.
- Platforms
mac· macOSwin· Windows- Applications
- Word · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsmulti· Cross-app- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Supplier revisions: 6 accepted, 3 rejected (text equals gold); 2 counters present as w:ins/w:del by 'Customer Counsel' with the issues-list language and anchored comments; no supplier revisions remain. VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (app metadata and rsids present).
Proposed partial credit
40% accept/reject outcomes; 40% counters tracked correctly; 20% comments and clean state.
R10.02.01.02-T05 Extract SPA deal points into a term sheet table
Original instruction
input/spa_execution_version.docx (synthetic share purchase agreement, 74 pages). Fill input/deal_points_template.xlsx with purchase price, adjustment mechanism, escrow amount and release date, indemnity basket type and amount, cap, survival periods (general and fundamental), non-compete term, and governing law, each with its section reference.
- Platforms
mac· macOSwin· Windows- Applications
- Word · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsmulti· Cross-app- Work product
WP02· Records/table- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Cell values vs curated key (numbers and durations exact; basket type exact) and section references exact; template layout preserved. Deterministic match on normalized values.
Proposed partial credit
60% deal-point values; 30% section references; 10% template intact.
R10.02.01.02-T06 Mark up a 40-page MSA against a twelve-point client playbook
Original instruction
input/vendor_msa.docx (40 pages) and input/playbook.pdf (12 positions: cap, consequential damages, data protection, audit, termination for convenience, notice periods, assignment, governing law, etc.). Mark up the MSA with Track Changes as 'Client Counsel' to meet every playbook position, keeping auto-numbering and cross-references intact, with a comment citing the playbook item for each change.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsnav· Pan/zoom/scrollclick· Point & click- Work product
WP01· Document- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Per playbook item, the clause's post-acceptance text satisfies a deterministic check (e.g. cap = 12 months fees, notice = 30 days, NY law); edits tracked by 'Client Counsel' with anchored comments; numbering list structure and REF fields resolve; LLM rubric only for drafting quality (15%). VBA macros and AppleScript are disabled by Office/OS policy and OOXML libraries are absent; the file must be saved by Word (rsids present).
Proposed partial credit
55% playbook positions met; 15% changes tracked with comments; 15% numbering/cross-refs intact; 15% drafting (LLM rubric).
R10.02.01.02-T07 Run a Litera Compare of two PDFs and export the change report
Original instruction
input/lease_v3.pdf and input/lease_v4.pdf are landlord drafts. In Litera Compare, compare v3 (original) to v4 (modified), export the redline as output/lease_v3_v4_redline.pdf and the change summary report as output/lease_changes.xlsx.
- Platforms
win· Windowsmac· macOS- Applications
- Litera Compare
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
click· Point & click- Work product
WP01· Document- Conditions
licensed· Licensed pro app- Proposed difficulty
- 2 / 5
Proposed verifier
Change report rows map to the gold diff (insertions/deletions/moves F1 >= 0.95); redline PDF text contains every gold change with Litera markup colors. Litera's command-line compare is disabled and other diff tools are absent from the VM; both files must be produced by Litera (producer metadata).
Proposed partial credit
50% change report; 40% redline PDF; 10% correct original/modified order.
R10.02.01.02-T08 Combine three reviewers' copies into one document
Original instruction
Three partners commented and edited copies of input/jv_term_sheet.docx (input/rev_AB.docx, rev_CK.docx, rev_LM.docx). Use Word's Combine to merge all revisions and comments into output/jv_term_sheet_combined.docx, preserving each reviewer as a distinct author; where two reviewers edited the same sentence, keep both revisions visible.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scroll- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Revision XML contains all revisions from the 3 copies (set equality by author+text) and all 9 comments with authors; conflicting sentence has both authors' revisions; base text unchanged. VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (app metadata and rsids present).
Proposed partial credit
50% revisions combined; 30% comments with authors; 20% conflicts preserved.
R10.02.01.02-T09 Find untracked changes slipped into a 'clean' counterparty draft
Original instruction
The counterparty returned input/apa_v5_clean.docx claiming it only accepts our v4 redline (input/apa_v4_redline.docx). Compare against v4 with all our changes accepted, identify every change not in our redline, and produce output/apa_v5_check.docx: a redline showing only those unflagged changes, each with a comment 'Untracked change - not agreed'.
- Platforms
mac· macOSwin· Windows- Applications
- Word · Litera Compare
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrolltype· Text & forms- Work product
WP01· Document- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 4 / 5
Proposed verifier
Redline revisions equal the 5 seeded untracked changes (including a number change in a schedule table and a moved sentence) with F1 = 1.0 for pass; one comment per change; no revisions for agreed edits. VBA macros and AppleScript automation are disabled by Office/OS policy, Litera's command-line compare is disabled and OOXML/diff libraries are absent; the redline must be saved by Word or Litera (app metadata).
Proposed partial credit
60% all hidden changes found; 20% no false positives; 20% comments.
R10.02.01.02-T10 Edit a lease in suggestion mode and resolve tenant comments
Original instruction
Google Docs (sandbox workspace) document 'Retail Lease - Unit 4B' has 7 open comments from the tenant. In Suggesting mode, make the edits the comments request when input/landlord_positions.pdf permits them, reply to the others with the landlord's position, and resolve only the comments you addressed with suggestions.
- Platforms
web· Any OS (web)- Applications
- Google Docs
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & click- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Docs export (docx with suggestions + comments): suggestions for the 4 permitted requests with correct text; replies on the 3 refused; exactly those 4 comments resolved. Docs/Drive API credentials are withheld from the agent, and revision history must show the edits from the browser session.
Proposed partial credit
45% suggestions; 30% replies; 25% correct resolution set.
R10.02.01.02-T11 Rename a defined term throughout with tracked changes and fix cross-references
Original instruction
Client wants 'Supplier' renamed 'Vendor' in input/services_agmt.docx, including headers, footers, tables, footnotes and the definitions section (keep alphabetical order of definitions). Make all changes tracked as 'Client Counsel', and make sure clause cross-references still point to the right sections after the definitions list is reordered.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsnav· Pan/zoom/scrollkey· Shortcuts- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
After accepting revisions: 0 'Supplier' left in all story parts (body, headers, footers, footnotes, tables), 'Vendor' definition in alphabetical position; revisions tracked; every REF field resolves to the gold target paragraph. VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (rsids present).
Proposed partial credit
45% all occurrences in every story; 20% definition order; 20% tracked; 15% cross-refs.
R10.02.01.02-T12 Abstract five scanned executed contracts with handwritten amendments
Original instruction
input/executed/ has 5 scanned signed vendor contracts (synthetic), some with handwritten, initialed changes to terms. Fill output/contract_abstracts.xlsx: effective date, initial term, renewal, termination notice days, assignment and change-of-control restrictions, liability cap, governing law, using the handwritten change where initialed by both parties and noting the page.
- Platforms
mac· macOSwin· Windows- Applications
- Adobe Acrobat Pro · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentnav· Pan/zoom/scrolltype· Text & forms- Work product
WP02· Records/table- Conditions
multiapp· Multi-applong· Long-horizonlicensed· Licensed pro app- Proposed difficulty
- 5 / 5
Proposed verifier
40 cells vs curated key (dates, numbers exact; restrictions as coded categories); the 4 seeded handwritten changes must be reflected only where initialed by both (1 is initialed by one side and must be ignored); page notes exact. OCR engines and Python spreadsheet libraries are absent from the VM, so scans must be read in Acrobat and the abstract saved by Excel.
Proposed partial credit
55% cell accuracy; 30% handwritten changes handled; 15% page references.
R10.02.01.02-T13 Compare with formatting noise suppressed to show substantive changes
Original instruction
input/credit_agmt_v6.docx and _v7.docx differ in fonts, styles, whitespace and hundreds of formatting tweaks plus 9 substantive changes. In Litera Compare, configure the comparison to ignore formatting, case and whitespace changes and detect moves, then export output/credit_v6_v7_substantive.docx (redline) and save the settings as a named profile 'Substantive only'.
- Platforms
win· Windowsmac· macOS- Applications
- Litera Compare
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
click· Point & clicknav· Pan/zoom/scroll- Work product
WP01· Document- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Redline revisions equal the 9 gold substantive changes (F1 >= 0.95) with move markup for the relocated clause; no formatting revisions; profile 'Substantive only' exists in Litera settings with the 3 ignore options. Litera's command-line compare is disabled and other diff tools are absent from the VM; the redline must be saved by Litera (producer metadata).
Proposed partial credit
55% substantive changes only; 20% move detected; 25% saved profile.
R10.02.01.02-T14 Accept only formatting revisions and keep content revisions open
Original instruction
input/lease_mixed_revisions.docx has 23 tracked revisions: 15 formatting-only and 8 insertions/deletions. Using the Review pane's markup filters, accept all formatting revisions and leave every insertion and deletion pending. Save as output/lease_formatting_accepted.docx.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & click- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 1 / 5
Proposed verifier
0 w:rPrChange/w:pPrChange remain; all 8 w:ins/w:del remain with original authors and dates; resulting formatting equals gold. VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (app metadata and rsids present).
Proposed partial credit
50% formatting accepted; 50% content revisions untouched.
R10.02.01.02-T15 Update a pricing schedule table with tracked cell edits
Original instruction
input/msa_schedule_b.docx contains Schedule B pricing (a 14-row Word table). Apply the agreed new rates from input/new_rates.xlsx with Track Changes as 'Vendor Counsel': change rates in place, add 2 new service rows, delete the discontinued row, and update the total formula field. Leave the rest of the document untouched.
- Platforms
mac· macOSwin· Windows- Applications
- Word · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & clickmulti· Cross-app- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Table after accepting revisions equals gold (rates exact, rows added/removed); revisions present as tracked cell changes, row insertion and row deletion by 'Vendor Counsel'; total field updated; text outside Schedule B unchanged. VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (rsids present).
Proposed partial credit
45% rates correct; 25% row insert/delete tracked; 15% total; 15% no collateral changes.
R10.02.01.02-T16 Check that a signed PDF conforms to the final agreed draft
Original instruction
input/signed_dpa.pdf (scanned signed version) must match input/dpa_final_agreed.docx. Compare them using Draftable (web) or Litera Compare after OCR, and list every discrepancy beyond signature blocks in output/conformity.json (section, agreed text, signed text). If none, write an empty list.
- Platforms
web· Any OS (web)win· Windows- Applications
- Draftable · Litera Compare · Adobe Acrobat Pro
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP12· Answer/decision- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Discrepancy list vs the 3 seeded differences (a changed day count, a dropped sentence, a changed party name) F1 = 1.0 for pass; OCR noise must not be reported (precision check). Python, standalone OCR engines and diff tools are absent from the VM and the Draftable API key is withheld, so the comparison must run in Draftable, Litera or Acrobat.
Proposed partial credit
70% seeded discrepancies found; 30% no OCR false positives.
R10.02.01.02-T17 Build a closing checklist from a merger agreement
Original instruction
input/merger_agreement.docx (90 pages, synthetic). Build output/closing_checklist.docx from input/checklist_template.docx: every closing condition, closing deliverable and pre-closing covenant with section reference, responsible party (Buyer/Seller/Both), and whether it is waivable, grouped by party, with a status column blank.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsmulti· Cross-app- Work product
WP01· Document- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Checklist table rows vs curated key (54 items): recall >= 0.9 matched by section reference, responsible party and waivable flag exact on matched rows, grouping correct; template styles preserved.
Proposed partial credit
45% item recall; 25% responsible party; 15% waivable flags; 15% grouping and template.
R10.02.01.02-T18 Reply to fifteen counterparty margin comments and make the agreed edits
Original instruction
input/sow_with_comments.docx has 15 margin comments from the counterparty. Using input/our_answers.xlsx, reply to each comment in its thread, make the tracked edits for the 6 items we agree to, mark those 6 comment threads resolved, and leave the rest open.
- Platforms
mac· macOSwin· Windows- Applications
- Word · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & clickmulti· Cross-app- Work product
WP01· Document- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
commentsExtended.xml: 15 threads each with our reply (text matches answers), exactly the 6 agreed threads marked done; 6 tracked edits at the right places with gold text; nothing else changed. VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (app metadata and rsids present).
Proposed partial credit
35% replies; 40% agreed edits tracked; 25% correct threads resolved.
R10.02.01.02-T19 Apply the same accept/reject decisions by editing revision XML (control)
Original instruction
Control pair of T02. Using only scripts (python-docx/lxml), process input/nda_redline.docx: accept all tracked changes except the Section 11 governing-law change, reject that one, and add the same comment on Section 11. Save output/nda_v3_cli.docx.
- Platforms
lin· Linux- Applications
- Python · lxml
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
code· Code & commands- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Same text and comment checks as T02; the file must open without repair in LibreOffice (headless conversion succeeds). Measures the GUI-vs-XML gap.
Proposed partial credit
60% correct text; 20% no revisions left; 20% comment anchored.
R10.02.01.02-T20 Delete a clause with tracked changes and repair numbering and cross-references
Original instruction
In input/license_agmt.docx, delete Section 7.3 (Most Favored Customer) with Track Changes as 'Licensor Counsel'. Subsequent sections must renumber automatically, every cross-reference to Sections 7.4-7.9 and to 7.3 itself must be updated or removed (with tracked edits), and the table of contents refreshed.
- Platforms
mac· macOSwin· Windows- Applications
- Word
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & clicktype· Text & forms- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
After accepting revisions: 7.3 text absent; numbering list continuous (7.3..7.8); all REF fields resolve to gold targets; the 2 references to old 7.3 removed or rephrased as tracked edits; TOC entries match headings. VBA macros and AppleScript automation are disabled by Office/OS policy and OOXML libraries are absent from the VM; the file must be saved by Word (rsids present).
Proposed partial credit
25% clause deleted tracked; 25% numbering; 35% cross-references; 15% TOC.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.