Clinical reasoning in simulated encounters
Work up a simulated patient and reach a diagnosis (control class).
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- AgentClinic · simulated EHR
- Class GUI-need label
AP· API-sufficient
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R09.01.01.04-T01 Diagnose a simulated patient through history and exam requests
Original instruction
Through the AgentClinic-style API (endpoints /ask_patient, /request_exam, /request_test, /diagnose), work up synthetic case MQ-0412: interview the patient agent, request physical exam findings and up to 5 tests, then submit one final diagnosis. Maximum 20 turns.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Final diagnosis mapped to SNOMED CT via the case's accepted-synonym list must match gold; turn and test limits enforced by the server; transcript stored for audit.
Proposed partial credit
20% key history elicited (3 gold items); 20% relevant exam requested; 60% correct diagnosis
R09.01.01.04-T02 Work up a complex case with image requests and test budget
Original instruction
Work up synthetic NEJM-style case NJ-077 via the AgentClinic API: the measurement agent returns lab values and, on request, a chest CT image (PNG) that you must interpret yourself. Submit a primary diagnosis and the 3 most important supporting findings within 25 turns.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & formsvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Diagnosis exact (synonym list); supporting findings matched to the case's gold finding list (2 of 3 must match); image request present in the transcript.
Proposed partial credit
15% image requested; 25% supporting findings; 60% diagnosis
R09.01.01.04-T03 Use chart history in a simulated EHR GUI to reach a diagnosis
Original instruction
In the simulated EHR web app (test login in input/creds.txt) open synthetic patient 'Rivera, Ana' and her current ED encounter. Review prior labs, medications and notes in the chart tabs, use the in-app 'Ask patient' panel, and submit the working diagnosis in the encounter's Assessment field.
- Platforms
web· Any OS (web)- Applications
- simulated EHR
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrolltype· Text & forms- Work product
WP12· Answer/decision- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 3 / 5
Proposed verifier
Assessment field diagnosis matches gold (drug-induced hyponatremia from a medication started 3 weeks earlier, visible only in the med history tab); app audit log must show the med-history tab was opened.
Proposed partial credit
25% relevant chart sections opened; 15% key question asked; 60% correct diagnosis
R09.01.01.04-T04 Reach a diagnosis with a distrustful simulated patient
Original instruction
Case PD-219 (pediatric fever, synthetic) via AgentClinic API: the parent agent is configured with a distrust bias and withholds the travel history unless asked respectfully and specifically. Elicit the history, request at most 4 tests, and submit a diagnosis.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Diagnosis exact (malaria per gold); transcript must contain the travel history disclosure; test count <=4 enforced.
Proposed partial credit
30% travel history elicited; 10% appropriate test chosen; 60% diagnosis
R09.01.01.04-T05 Diagnose under a cost budget for tests
Original instruction
Case CB-033 via the AgentClinic API with a priced test catalog (GET /tests). You may spend at most $500 and order at most 3 tests. Submit the diagnosis plus your total spend.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Server enforces budget; score = diagnosis correctness x (1 - 0.3 * spend/500); unnecessary high-cost imaging is penalized via spend.
Proposed partial credit
60% correct diagnosis; 25% within budget; 15% spend below $250
R09.01.01.04-T06 Assign ESI triage levels to ten simulated ED arrivals
Original instruction
The simulated ED API streams 10 synthetic arrivals with chief complaint, vitals and brief history (GET /arrivals). For each, ask up to 2 clarifying questions and assign an Emergency Severity Index level 1-5 via POST /triage.
- Platforms
web· Any OS (web)- Applications
- simulated EHR
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Per-patient ESI vs expert consensus: exact match scored 1, off by one 0.5; under-triage of ESI-1/2 patients scored 0 and flagged.
Proposed partial credit
80% ESI accuracy (weighted); 20% no under-triage of high-acuity patients
R09.01.01.04-T07 Produce a calibrated differential diagnosis for a dyspnea case
Original instruction
For case DX-150 via the AgentClinic API, gather history and up to 4 tests, then submit a top-5 differential with probabilities summing to 1 (POST /differential) instead of a single diagnosis.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Gold diagnosis in top 3 required for full credit; multiclass Brier score vs gold one-hot computed; probabilities must sum to 1 +-0.01.
Proposed partial credit
40% gold in top-3; 20% gold ranked first; 40% Brier score scaled
R09.01.01.04-T08 Choose and renally dose an antibiotic in a simulated encounter
Original instruction
Case RX-041 (synthetic, elderly patient with pyelonephritis, eGFR and weight in the chart via GET /chart). Confirm the diagnosis, check allergies, compute creatinine clearance (Cockcroft-Gault), and submit an antibiotic order (drug, dose, route, interval) via POST /order using input/renal_dosing.pdf.
- Platforms
web· Any OS (web)- Applications
- simulated EHR
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Drug in the acceptable set and not contraindicated by the documented allergy; dose and interval match the CrCl band in the reference table; CrCl within 5 mL/min reported.
Proposed partial credit
20% diagnosis confirmed; 20% allergy respected; 20% CrCl; 40% dose and interval
R09.01.01.04-T09 Interview a Spanish-speaking simulated patient and diagnose
Original instruction
Case ES-062 via the AgentClinic API: the patient agent answers only in Spanish. Conduct the history in Spanish, request exams and up to 4 tests, and submit the diagnosis plus a 2-sentence English summary of the key history.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Diagnosis exact; summary checked for the 4 gold history facts by keyword/semantic match (LLM judge stated, 20% weight).
Proposed partial credit
15% history conducted in Spanish; 25% key facts in summary; 60% diagnosis
R09.01.01.04-T10 Escalate a red-flag chest pain case instead of outpatient workup
Original instruction
Case CP-008 is framed as a telehealth visit for 'heartburn'. Take the history via the API. If findings indicate a possible acute coronary syndrome, the correct action is POST /escalate with reason, not an outpatient diagnosis or prescription.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Escalation call made within 10 turns with a reason mentioning ACS/cardiac cause; no prescription or discharge action before it; key risk factors elicited.
Proposed partial credit
30% red-flag history elicited; 50% escalation with correct reason; 20% no unsafe action
R09.01.01.04-T11 Review diabetes control in a simulated EHR and document a plan
Original instruction
In the simulated EHR web app open synthetic patient 'Okafor, James' follow-up visit. Review the A1c trend, current medications and eGFR, then fill the structured Plan form: medication change (drug, dose), labs to order, and follow-up interval, consistent with input/diabetes_guideline.pdf.
- Platforms
web· Any OS (web)- Applications
- simulated EHR
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrolltype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Plan form fields checked against the guideline-derived acceptable set: intensification chosen appropriately (metformin dose capped by eGFR), labs include A1c and urine ACR, follow-up 3 months.
Proposed partial credit
40% medication decision; 30% labs; 15% follow-up; 15% chart sections reviewed
R09.01.01.04-T12 Diagnose a dermatology case from a requested lesion photo
Original instruction
Case DM-031 via the AgentClinic API: request the lesion photograph (returned as an image) and dermoscopy description, ask relevant history, and submit a diagnosis and next step (reassure, biopsy, refer).
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & formsvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Diagnosis exact (synonym list); next step exact (excisional biopsy for suspected melanoma); photo request present.
Proposed partial credit
15% image requested; 55% diagnosis; 30% next step
R09.01.01.04-T13 Ask for the missing exam before diagnosing an incomplete case
Original instruction
Case IN-005 (synthetic headache) via the API deliberately withholds the neurological exam and fundoscopy unless requested. Do not diagnose until you have the information needed to exclude raised intracranial pressure; then submit the diagnosis.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
clarify· Ask when unclear- Proposed difficulty
- 2 / 5
Proposed verifier
Transcript must show fundoscopy and neuro exam requested before /diagnose; diagnosis exact; diagnosing without them scores at most 30%.
Proposed partial credit
40% required exams requested first; 60% diagnosis
R09.01.01.04-T14 Revise a diagnosis at a follow-up visit with new results
Original instruction
Case FU-018 has two encounters. In visit 1 (API session A) make a working diagnosis and order tests. In visit 2 (session B, two weeks later) the API returns the test results and new symptoms; revise the diagnosis and submit a management plan.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 4 / 5
Proposed verifier
Visit 1 tests must include the discriminating test; visit 2 diagnosis exact; plan contains the 2 gold management elements (keyword/semantic match).
Proposed partial credit
20% visit-1 working diagnosis reasonable; 25% discriminating test ordered; 35% final diagnosis; 20% plan
R09.01.01.04-T15 Compute stroke and bleeding risk scores from a simulated chart
Original instruction
In the simulated EHR web app, open synthetic patient 'Lindqvist, Eva' with new atrial fibrillation. From the problem list, vitals and labs compute CHA2DS2-VASc and HAS-BLED, enter them in the Risk Scores form, and select the anticoagulation decision consistent with input/af_guideline.pdf.
- Platforms
web· Any OS (web)- Applications
- simulated EHR
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrolltype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Saved form: CHA2DS2-VASc and HAS-BLED exact; decision in the guideline-acceptable set (DOAC recommended); audit log shows the problem list and labs were viewed.
Proposed partial credit
35% CHA2DS2-VASc; 35% HAS-BLED; 30% decision
R09.01.01.04-T16 Administer PHQ-9 and act on suicide-risk screening in dialogue
Original instruction
Case PS-024 via the API: a primary-care patient reports low mood. Administer the PHQ-9 conversationally, score it, and if item 9 is positive complete the risk screen in input/csrss_screener.pdf and take the appropriate action (POST /safety_plan or /escalate).
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
PHQ-9 total exact (from the patient agent's hidden answers); severity category exact; risk-screen outcome triggers the correct action per screener rules (escalate for recent plan).
Proposed partial credit
30% PHQ-9 total; 15% severity; 20% screener completed; 35% correct safety action
R09.01.01.04-T17 Recognize sepsis from flowsheet trends in a simulated EHR
Original instruction
In the simulated EHR web app, review synthetic inpatient 'Haddad, Omar' flowsheets over the last 24 h (vitals, lactate, WBC, urine output). Identify the time sepsis criteria were first met, document it in the note, and place the sepsis bundle orders in the correct order with timestamps.
- Platforms
web· Any OS (web)- Applications
- simulated EHR
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP08· App/system state- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 5 / 5
Proposed verifier
Documented onset time within 1 h of gold; orders contain lactate, blood cultures before antibiotics, broad-spectrum antibiotic and 30 mL/kg fluids with correct volume for weight; order sequence checked.
Proposed partial credit
30% onset time; 20% cultures before antibiotics; 30% bundle orders; 20% fluid volume
R09.01.01.04-T18 Diagnose a rare multisystem disease with targeted genetic testing
Original instruction
Case RD-003 via the AgentClinic API presents with recurrent fevers, rash and arthralgia over years. Gather family history and labs, request at most one genetic panel from the catalog, and submit the diagnosis and the confirming gene.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Diagnosis exact; gene exact; only one genetic test allowed (server-enforced); family-history elicitation required in transcript.
Proposed partial credit
20% family history; 20% correct panel chosen; 40% diagnosis; 20% gene
R09.01.01.04-T19 Apply a hypertension guideline to a simulated outpatient
Original instruction
Case HT-012 via the API: gather blood pressure readings, risk factors and current medications, then submit a management decision per input/htn_guideline.pdf: diagnosis stage, whether to start or intensify medication, which drug class, and recheck interval.
- Platforms
web· Any OS (web)- Applications
- AgentClinic
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
type· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Stage exact; start/intensify decision exact; drug class in the acceptable set given comorbidities; recheck interval exact.
Proposed partial credit
25% stage; 25% decision; 30% drug class; 20% interval
R09.01.01.04-T20 Manage three simulated patients concurrently with interrupting pages
Original instruction
In the simulated ward app (web, test login), you cover 3 synthetic inpatients for a 2-hour simulated shift. Pages arrive asynchronously (new chest pain, low potassium result, family question). Prioritize, examine via the app, place orders and document; the clock runs at 10x.
- Platforms
web· Any OS (web)- Applications
- simulated EHR
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsrt· Real-time control- Work product
WP08· App/system state- Conditions
interrupt· Interruptionslong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Event log scored: time-to-response for the chest pain page <10 sim-min with ECG and troponin ordered; potassium replacement dose correct; no unsafe orders; documentation present for each patient.
Proposed partial credit
35% chest pain response; 25% potassium management; 20% family question handled; 20% documentation and no unsafe orders
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.