Permits, licensing & records processing
Review applications and update permit and records systems.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- permitting systems (sandbox)
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R10.03.01.03-T01 Take in a residential fence permit application
Original instruction
Accela Civic Platform sandbox (city of 'Fairview', synthetic). From input/fence_application.pdf create a Residential Fence permit record: applicant and owner contacts, parcel by APN, fence height, material and length, valuation, and the plot plan attachment. Assess the fee from the configured schedule and set the record status to 'Received'.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & click- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Sandbox record fields equal gold (APN linked to the parcel object, ASI fields exact), attachment present, invoiced fee equals schedule, status Received. Construct API disabled for the agent user; record history shows citizen/staff UI actions.
Proposed partial credit
40% record fields; 20% parcel link; 20% fee; 20% attachment and status.
R10.03.01.03-T02 Run a completeness review and send an incomplete-application notice
Original instruction
Accela sandbox, record BLD-26-00412 (single-family addition). Compare the submitted documents with input/submittal_checklist.pdf, mark each checklist item complete or deficient in the record's checklist, set status 'Incomplete', and generate the incomplete notice listing every missing or deficient item from the notice template.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmenttype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Checklist item states equal gold (4 deficient incl. an unsigned owner affidavit and a structural sheet missing the engineer stamp); status Incomplete; generated notice lists exactly those 4 items. Accela Construct API tokens are withheld and EMSE scripting is denied to the agent's role, so record audit history must show the changes from the web session.
Proposed partial credit
50% checklist states; 20% status; 30% notice content.
R10.03.01.03-T03 Compute and invoice building permit fees from a valuation table
Original instruction
Accela sandbox, record BLD-26-00418 (commercial alteration, valuation in input/valuation.pdf). Using input/fee_schedule.pdf compute the building permit fee from the valuation brackets, plan review fee (65% of permit fee), technology surcharge and state fee, add them as fee items with the right codes, and invoice them.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & click- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Fee items and amounts exact to the cent (bracket math and rounding per schedule); fee codes exact; all invoiced; no duplicate items. Accela Construct API tokens are withheld and EMSE scripting is denied to the agent's role, so record audit history must show the fees added from the web session.
Proposed partial credit
45% permit fee; 20% plan review; 20% surcharges; 15% invoiced without duplicates.
R10.03.01.03-T04 Mark up a plan set with code comments and measured checks in Bluebeam Revu
Original instruction
input/ti_plans.pdf (12-sheet tenant improvement plan set). In Bluebeam Revu calibrate each floor-plan sheet to its stated scale, measure the two exit corridor widths and the stair riser height in the section, and place cloud+callout markups with comments at each code deficiency on input/review_checklist.pdf (subject 'Building', status 'Open'). Save output/ti_plans_reviewed.pdf.
- Platforms
win· Windows- Applications
- Bluebeam Revu
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
geom· Precise geometryvis· Visual judgmentclick· Point & click- Work product
WP10· Annotations- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
PDF markups parsed: calibration scale per sheet correct; length measurements within 1% of gold; cloud markups overlap the 7 gold deficiency regions (IoU >= 0.3) with matching comment codes; subject/status set. Bluebeam scripting and batch automation are disabled and PDF libraries are absent from the VM; measurement annotations must carry Bluebeam's /Measure dictionaries.
Proposed partial credit
20% calibration; 30% measurements; 40% deficiency clouds and comments; 10% markup properties.
R10.03.01.03-T05 Verify parcel zoning and setbacks in the city GIS and record the determination
Original instruction
For record PLN-26-0090 (accessory dwelling unit) in the Accela sandbox, open the city's ArcGIS Online parcel viewer (sandbox), identify the parcel's zoning district and any overlay, read setbacks from input/zoning_code.pdf, and record zoning, overlay and the required front/side/rear setbacks in the record's zoning-review fields, with 'Conforms' or 'Does not conform' for the proposed site plan.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox) · ArcGIS Online
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & clickmulti· Cross-app- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Zoning, overlay and setbacks equal gold (parcel sits on a seeded overlay boundary only visible on the map); conformity determination equals gold computed from site-plan dimensions. ArcGIS REST queries and Accela Construct API tokens are blocked for the agent, so the overlay must be read in the map viewer and fields entered in the web session (audit history).
Proposed partial credit
35% zoning and overlay; 35% setbacks; 30% conformity.
R10.03.01.03-T06 Carry a commercial permit through multi-discipline review to issuance
Original instruction
Accela sandbox, record BLD-26-00431 (restaurant TI). Route plan review to Building, Fire, Health and Planning, enter each discipline's comments from input/review_comments/, issue the consolidated correction letter, process the resubmittal in input/resubmittal/ against the comments, approve with the 3 conditions listed in input/conditions.txt, collect the test-payment fees and issue the permit in the sandbox.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
long· Long-horizonrisk· Irreversible/risky- Proposed difficulty
- 5 / 5
Proposed verifier
Workflow history equals gold sequence (4 reviews, corrections, resubmittal, approvals); comments recorded per discipline; correction letter generated; conditions attached; fees paid via test processor; status Issued with permit PDF generated. Accela Construct API tokens are withheld and EMSE scripting is denied, so each step must appear in audit history from the web session.
Proposed partial credit
25% routing and comments; 20% correction letter; 25% resubmittal review; 30% conditions, fees and issuance.
R10.03.01.03-T07 Record a failed framing inspection and schedule the re-inspection
Original instruction
Accela sandbox, record BLD-26-00377. Result today's framing inspection as Failed with the 3 correction items from input/inspector_notes.txt (checklist items and comments), then schedule a re-inspection on the next available slot for the same inspector, respecting the 24-hour notice rule.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Inspection result Failed with 3 checklist deficiencies; new inspection scheduled for the gold slot (first slot >= 24 h later on the inspector's calendar); same inspector. Accela Construct API tokens are withheld and EMSE scripting is denied to the agent's role, so the result and schedule must appear in audit history from the web session.
Proposed partial credit
50% result and deficiencies; 50% re-inspection scheduled correctly.
R10.03.01.03-T08 Renew a business license with insurance verification and a late penalty
Original instruction
Tyler EnerGov sandbox, business license BL-2025-1187 (food truck). The renewal came in 17 days late with input/coi.pdf. Verify the insurance certificate's insured name, coverage limits and expiry against the license requirements, enter the new policy, compute the renewal fee plus the late penalty from input/license_fees.pdf, and issue the renewed license certificate if all requirements are met.
- Platforms
web· Any OS (web)- Applications
- Tyler EnerGov (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Insurance record fields exact; fee and penalty exact; license status Active with new expiry date; certificate PDF generated. If gold says the COI fails (seeded variant), license must stay pending with a deficiency note. EnerGov API credentials are withheld from the agent, so changes must appear in the license audit history from the web session.
Proposed partial credit
30% insurance verification; 35% fees; 35% correct issuance decision and certificate.
R10.03.01.03-T09 Fulfill a public records request with exemptions and a release log
Original instruction
NextRequest-style records portal (sandbox). Request R26-0331 asks for code-enforcement records for 14 Elm St (synthetic) in 2025. Compute the statutory response deadline from input/records_act.pdf (business days, holidays), find responsive documents in the repository, redact exempt data (SSNs, phone numbers of complainants, personnel notes) in Acrobat Pro, upload releases with an exemption log citing statute sections, and close the request.
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- NextRequest (sandbox) · Adobe Acrobat Pro
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollgeom· Precise geometrymulti· Cross-app- Work product
WP08· App/system state- Conditions
multiapp· Multi-apprisk· Irreversible/riskylicensed· Licensed pro app- Proposed difficulty
- 5 / 5
Proposed verifier
Due date exact; released set equals the 9 gold responsive files; OCR/text of releases contains none of the seeded exempt strings, complainant identity removed; exemption log rows cite the right sections; request closed within due date. NextRequest API tokens are withheld and PDF libraries are absent, so redaction must be done in Acrobat and uploads appear in the request log from the web session.
Proposed partial credit
15% deadline; 30% responsive set; 35% redactions; 20% exemption log and closure.
R10.03.01.03-T10 Verify a contractor's state license during permit intake
Original instruction
Accela sandbox, record BLD-26-00440 lists contractor license number 1048832. Look it up on the state licensing board's lookup page (sandbox), check classification, status, expiry and workers' comp coverage against input/contractor_rules.pdf, and either link the licensed professional to the record or put the record on hold with a deficiency explaining the problem.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox) · state license lookup (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsmulti· Cross-app- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Decision equals gold (seeded: license active but workers' comp expired); record on hold with a deficiency naming workers' comp; LP not linked. Variant seeds check the positive path. Accela Construct API tokens are withheld and EMSE scripting is denied to the agent's role, so the hold must appear in audit history from the web session.
Proposed partial credit
40% lookup facts recorded; 40% correct decision; 20% deficiency text.
R10.03.01.03-T11 Consolidate three reviewers' markups into one comment letter in a Studio Session
Original instruction
Bluebeam Revu Studio Session 'BLD-26-00431 Review 1' (sandbox) holds markups from Building, Fire and Planning reviewers on the plan set. Resolve duplicates (same issue noted twice), set each markup status Accepted or Rejected per input/supervisor_decisions.csv, export the markups list as output/comment_letter.pdf with columns Sheet, Discipline, Comment, Status, and finish the session.
- Platforms
win· Windows- Applications
- Bluebeam Revu · Bluebeam Studio
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP08· App/system state- Conditions
licensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
Session markups: duplicates merged (count equals gold), statuses equal decisions; exported markups list rows equal gold with the 4 columns; session closed with final PDF. Status changes only exist in the session store; the agent has no Bluebeam Studio API (OAuth) credentials and Revu scripting/batch automation is disabled, so markups must be changed in Revu.
Proposed partial credit
30% duplicates resolved; 35% statuses; 25% exported letter; 10% session finished.
R10.03.01.03-T12 Index sixty scanned historical permits into the records system
Original instruction
input/scans_1978_1985/ holds 60 scanned paper building permits (synthetic). In the Accela sandbox, create a historical record for each with permit number, issue date, work type and APN, attach the scan, and link the parcel. Parcels later split or renumbered must be linked to the current parcels using the parcel history layer in the city's ArcGIS Online viewer.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox) · ArcGIS Online
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmenttype· Text & formsmulti· Cross-app- Work product
WP02· Records/table- Conditions
long· Long-horizonhidden· Hidden/stale statemultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
60 records with fields vs gold (>= 97% field accuracy; handwriting seeded); scans attached (hash); parcel links equal gold including the 7 split/renumbered parcels resolved via history. Accela Construct API tokens are withheld and EMSE scripting is denied to the agent's role, so records must appear in audit history from the web session.
Proposed partial credit
45% field accuracy; 15% attachments; 40% parcel links incl. history cases.
R10.03.01.03-T13 Open a code-enforcement case and issue a notice of violation
Original instruction
Accela sandbox, code enforcement module. From complaint input/complaint_0918.pdf and photos input/photos/, open a case on the right parcel, add violations from the municipal code list (overgrown vegetation, inoperable vehicle), attach photos, and generate the notice of violation with a compliance date 14 calendar days out.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
type· Text & formsclick· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Case linked to parcel; 2 violation records with correct code sections; photos attached; NOV document generated with compliance date = today + 14; complainant identity kept out of the NOV text. Accela Construct API tokens are withheld and EMSE scripting is denied to the agent's role, so the case must appear in audit history from the web session.
Proposed partial credit
25% case and parcel; 30% violations; 15% photos; 30% NOV correct.
R10.03.01.03-T14 Find when a parcel's last roof permit was finaled
Original instruction
In the Accela sandbox citizen portal, search permit history for 2207 Cedar Ave (synthetic) and determine the final-inspection date of the most recent re-roof permit and its permit number. Write output/answer.json.
- Platforms
web· Any OS (web)- Applications
- Accela Citizen Access (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 1 / 5
Proposed verifier
Permit number and final date equal the sandbox record; a later re-roof permit that expired without final is seeded as a distractor. Accela Construct API access is blocked from the agent VM, so permit history is reachable only through the Citizen Access web portal.
Proposed partial credit
50% permit number; 50% final date.
R10.03.01.03-T15 Decide whether a certificate of occupancy can be issued
Original instruction
Accela sandbox, record BLD-26-00290 (new duplex). Before issuing the certificate of occupancy, confirm every required inspection is passed, every approval condition is satisfied, and all fees are paid. If anything is outstanding, do not issue the CO; issue a temporary CO only if input/tco_policy.pdf allows it for that item, otherwise add a hold explaining what remains.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
risk· Irreversible/riskyhidden· Hidden/stale state- Proposed difficulty
- 4 / 5
Proposed verifier
Seeded state: one landscaping condition unmet (TCO-eligible) and all else complete; correct outcome is a TCO with expiry per policy and a condition note; issuing a full CO scores 0. Accela Construct API tokens are withheld and EMSE scripting is denied to the agent's role, so the TCO must appear in audit history from the web session.
Proposed partial credit
40% outstanding item found; 40% correct TCO vs hold decision; 20% expiry and note.
R10.03.01.03-T16 Draw a right-of-way work zone on the permit map and attach traffic control
Original instruction
OpenGov Permitting sandbox, encroachment permit ROW-26-0077. Using the application's map widget, draw the work-zone polygon along the curb lane of Main St between 3rd and 5th Ave as shown in input/work_zone_sketch.pdf, set the closure type and dates, and attach input/tcp.pdf as the traffic control plan.
- Platforms
web· Any OS (web)- Applications
- OpenGov Permitting (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathgeom· Precise geometrynav· Pan/zoom/scroll- Work product
WP04· 3D/spatial model- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Stored polygon vs gold geometry IoU >= 0.6 and within the right-of-way layer; closure type and dates exact; TCP attached. OpenGov API credentials are withheld from the agent, so the polygon must be drawn with the application's map widget and appear in record history from the browser session.
Proposed partial credit
55% polygon geometry; 25% closure fields; 20% attachment.
R10.03.01.03-T17 Void a withdrawn permit and process the partial refund
Original instruction
Accela sandbox, record BLD-26-00402 was withdrawn after plan review started. Per input/refund_policy.pdf, void the permit, compute the refundable amount (plan review fee retained in proportion to completed review stages), create the refund record with the right GL accounts in the sandbox, and generate the refund letter. Do not touch other records of the applicant.
- Platforms
web· Any OS (web)- Applications
- Accela Civic Platform (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Status Void with reason; refund amount exact; refund transaction lines with GL codes per policy; letter generated; applicant's other 2 records unchanged. Accela Construct API tokens are withheld and EMSE scripting is denied to the agent's role, so the void and refund must appear in audit history from the web session.
Proposed partial credit
20% voided; 40% refund amount; 25% GL lines; 15% letter and no collateral changes.
R10.03.01.03-T18 Decide twelve short-term rental license applications against zoning and caps
Original instruction
Tyler EnerGov sandbox: 12 pending short-term rental license applications. For each, check the parcel's zoning eligibility and whether the neighborhood's cap (input/str_caps.csv, counting active licenses) is reached, using the city's ArcGIS Online zoning overlay; approve or deny with the reason code, processing in submission order so caps fill correctly.
- Platforms
web· Any OS (web)- Applications
- Tyler EnerGov (sandbox) · ArcGIS Online
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollmulti· Cross-app- Work product
WP08· App/system state- Conditions
multiapp· Multi-applicensed· Licensed pro app- Proposed difficulty
- 4 / 5
Proposed verifier
Final status and reason per application equal gold (order-dependent cap filling); counts of active licenses per neighborhood never exceed caps. EnerGov API credentials are withheld and ArcGIS REST queries are blocked for the agent, so decisions must be made via the map viewer and appear in EnerGov audit history from the web session.
Proposed partial credit
40% zoning eligibility; 40% cap-aware decisions; 20% reason codes.
R10.03.01.03-T19 Create the same fence permit through the Accela Construct API (control)
Original instruction
Control pair of T01. With API credentials in env ACCELA_APP_ID/ACCELA_TOKEN on a fresh sandbox agency, create the Residential Fence record from input/fence_application.json with the same contacts, parcel, custom fields, attachment and fee assessment, and set status Received.
- Platforms
lin· Linux- Applications
- Accela Construct API · Python
- Interfaces
api· API/SDKterm· Terminal- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Same record checks as T01. Measures the GUI-vs-API gap on identical content.
Proposed partial credit
40% record fields; 20% parcel link; 20% fee; 20% attachment and status.
R10.03.01.03-T20 Record a deed and index it in a county land-records system
Original instruction
A county recorder land-records sandbox (Windows desktop client). Record input/warranty_deed.pdf: document type, grantors and grantees exactly as named, legal description (lot/block/subdivision), consideration, recording and transfer-tax fees per input/recording_fees.pdf, then stamp it with the assigned instrument number and save output/recorded_deed.pdf.
- Platforms
win· Windows- Applications
- land records system (sandbox)
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
type· Text & formsclick· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Index rows: grantor/grantee names and order exact, legal description fields exact, fees exact; instrument number sequential; stamped PDF shows the number and recording date. The desktop client has no API.
Proposed partial credit
40% party indexing; 25% legal description; 20% fees; 15% stamped PDF.
Mapped research references
No benchmark reference is assigned to this class in the current mapping.