Summary
Introduction
The rapid integration of Large Language Models (LLMs) into clinical workflows has highlighted a gap between passing textbook-style licensing exams and managing real-world patient trajectories safely. Traditional static benchmarks (like multiple-choice questions or single clinical vignettes) evaluate isolated factual recall rather than longitudinal reasoning.
Furthermore, existing frameworks often fail to account for:
Harms of omission: Failing to recommend a critical action or safety warning, which accounts for more than 75%–80% of severe medical AI errors.
Resource limitations: Clinical reasoning in real-world, resource-constrained environments (e.g., rural Indian healthcare settings).
Trajectory fidelity: The ability of an AI to maintain a coherent, mathematically and logically consistent representation of a patient's case across multiple touchpoints over time, without "forgetting," contradicting itself, or retrospectively rewriting past uncertainty.
This phase 1 study outlines a novel benchmarking methodology designed to evaluate longitudinal, resource-aware clinical reasoning using authentic Patient Journey Records (PaJRs) as live, real-world benchmarks.
Methods
The benchmark evaluates clinical reasoning via a prospective, stage-gated process:
Case Selection & Eligibility: Real-world PaJRs representing complex trajectories (spanning hours to months) are selected. To ensure adequate cognitive complexity, cases must feature:
At least five sequential clinical reasoning checkpoints with active "evidence-decision loops."
Full patient demographics, vital signs, two to five comorbidities with active medications, and realistic geographical/socioeconomic context.
Explicit resource constraints reflecting typical Indian healthcare settings (e.g., limited diagnostics, absence of immediate specialist care).
Case 1 (Baseline): A 50-year-old female with Russell's viper envenomation complicated by severe anti-snake venom (ASV) induced anaphylactic shock.
The ODAO & Trajectory Evaluation Framework:
Observation (O): Interpreting evolving clinical parameters and physiological/lab trends.
Diagnosis (D): Identifying the core clinical issues under dynamic uncertainty.
Action (A): Selecting management strategies, balancing patient safety against immediate localized resource limitations and therapeutic risks.
Outcome (O): Anticipating trajectories, complications, and long-term sequelae.
Assessment Structure:
Seven-item Multiple Choice Questions (MCQs) are constructed per case (incorporating systematic distractors).
Unanswerable Questions & Refusal Metrics: At least 1–2 questions per case are deliberately designed to be unanswerable with the information provided in the PaJR. This tests an LLM's epistemic restraint—its ability to safely refuse to answer or declare missing data, reflecting critical findings from the Stanford-Harvard NOHARM safety benchmark.
Trajectory Fidelity & Continuity: Tracks whether models maintain a coherent representation of patient data across multiple iterations, resisting the urge to fill evidentiary gaps with plausible-sounding hallucinations or retrospectively rewriting past diagnostics once the clinical outcome is revealed.
Evaluated Systems & Comparators: The intervention clinical system (implemented on Gemini 3.1 Flash Lite) is compared against frontier models including Claude 3.5 Sonnet, GPT-4o, Gemini 1.5/2.5 Pro, Gemini 3.5 Flash, and MedGemma.
Steering & Intervention Analysis: If a model makes an unsafe recommendation or a critical omission, subsequent human-in-the-loop interventions (prompts, corrections) are logged. The analysis measures Correction Persistence (whether the correction survives future turns) vs. Recursive Failure.
Results (Anticipated for Phase 1 Setup)
By transitioning the evaluation of clinical reasoning from static diagnostic endpoints to longitudinal pathways, the benchmark is designed to yield:
An Autonomous LLM Score (0–7) reflecting baseline prospective accuracy.
An Epistemic Restraint Rating measuring correct refusals when data is absent.
A Trajectory Fidelity Index mapping the model's ability to retain, update, and carry forward patient states without self-contradiction.
Steering Metrics that quantify the frequency and nature of necessary human intervention, distinguishing independent reasoning from retrospectively "steered" correctness.
Discussion
Evaluating clinical models using longitudinal PaJRs forces AI systems to operate in realistic environments where "correct" decisions are inextricably linked to systemic resource limits. Incorporating "refusal-to-answer" metrics addresses a critical, under-measured axis of AI safety.
A model's clinical utility is not merely determined by whether a patient ultimately recovers (outcome-based epistemic reversal), but by whether the cognitive process leading to the action was safe, robust, and reproducible under local constraints.
Ultimately, this benchmark aims to elevate local clinical care by matching global algorithmic performance to the real, unpolished, and constrained reality of local patient journeys.
Keywords
Clinical Reasoning Benchmarking
Patient Journey Records (PaJRs)
Observation-Diagnosis-Action-Outcome (ODAO) Framework
Trajectory Fidelity
Epistemic Restraint
Resource-Constrained Healthcare
Medical AI Safety
Harms of Omission
Human-in-the-Loop Steering
Russell's Viper Envenomation
Thematic Analysis
The transcript outlines an active methodological debate regarding how to test medical AI. The discussion resolves into four primary themes that illustrate the transition from global learning standards to local caring outcomes.
THEMATIC MAP
[Global Learning] [Local Caring]
Standardized AI ───────────────────► Grounded in Real-World
Benchmarks Resource Constraints
│ │
▼ ▼
[Epistemic Restraint] ───────────────────► [Trajectory Fidelity]
Knowing when to say Coherence across the entire
"I do not know" longitudinal journey
Theme 1: Shifting from Static Accuracy to Longitudinal Trajectory
The Concept: Standard medical AI evaluations are often "static"—they test if an AI can correctly identify a diagnosis from a complete, clinical textbook vignette. The transcript argues that real clinical care is a "journey" characterized by evolving uncertainty, shifting vitals, and diagnostic revision.
Grounded Evidence: hu1 outlines that the benchmark must utilize PaJRs that "document the complete trajectory of patient care, including evolving clinical observations, diagnostic uncertainty, therapeutic decisions... each PaJR case serves as a real world reasoning pathway rather than a single clinical vignette."
Workflow Evaluation: By forcing models to reason prospectively without knowing the final outcome, the framework evaluates whether global AI reasoning can mirror the step-by-step cognitive workflow of a local practitioner.
Theme 2: Capturing "Harms of Omission" and the Necessity of Refusal
The Concept: Multiple-choice questions often inadvertently hand-hold AI by displaying correct options on the screen. True clinical danger in the field rarely stems from choosing the wrong written option; instead, it stems from omission—forgetting to check a red flag, omitting a vital safety warning, or guessing when critical data is missing.
Grounded Evidence: hu1 notes that "We cannot catch the 'harms of omission' using mcqs alone... Phase 2 will point out if AI forgets to mention any critical safety warning, or miss out any red flag." Furthermore, hu4 links this to the Stanford ARISE group's NOHARM study and notes: "Atleast 1 question per case should be designed for this [refusal to answer]."
Workflow Evaluation: Evaluating an AI's willingness to say "I do not know" or "The record lacks sufficient data to safely decide" establishes epistemic restraint. In a local clinic, an AI that confidently guesses when it lacks key patient history is a severe safety liability.
Theme 3: Trajectory Fidelity vs. Cognitive Drift
The Concept: A major hazard of generative AI in complex workflows is "cognitive drift"—the tendency of a model to lose historical context, contradict its own prior assertions, or silently fill in missing data with highly plausible clinical hallucinations.
Grounded Evidence: hu6 explains that "A model could score well across seven MCQs while still losing prior facts, dropping unresolved uncertainty or constraints, contradicting itself... [We need] a trajectory fidelity layer: does the model preserve a coherent, evolving representation of the patient across time... and avoiding retrospective rewriting once the outcome is known?"
Workflow Evaluation: If an AI assistant cannot maintain "epistemic memory" (e.g., remembering that a patient's allergy or a localized resource constraint was established three steps ago), its global training fails to translate into a safe, locally coordinated plan of care.
Theme 4: Resisting "Outcome-Based Epistemic Reversal"
The Concept: In medicine, a good outcome does not automatically validate the clinical pathway that preceded it (e.g., a patient surviving a severely mismanaged snakebite due to sheer physiological resilience). Conversely, a poor outcome does not mean the clinical decisions made along the way were incorrect. AI benchmarks must evaluate the quality of reasoning, not just correlate outputs with final survival data.
Grounded Evidence: hu6 highlights the danger of this bias using the baseline case: "The viper case itself seems to show the danger of outcome-based epistemic reversal if later recovery is used to retrospectively redefine the original severity as mild."
Workflow Evaluation: By separating "outcome concordance" from "reasoning quality," the benchmark ensures that AI models are graded on their prospective clinical logic under resource limits—preventing the model from relying on hindsight bias to make decisions.
Provide an imrad summary, keywords and thematic analysis for the content below focusing on how one can evaluate medical cognition workflows toward global learning outcomes to achieve local caring outcomes.
Study design for phase 1 of Benchmarking Clinical Reasoning in Resource-Constrained Settings
Conversational transcripts:
[16/07, 12:36]hu4: The entire drill has to be completed on case 1 before we write the methods section, and then move on to the next case based on the finalized methods. Once methods are written and finalized, we shouldn't be changing them midway atleast for this manuscript. Hence my opinion is that we finalize the methods section first and write it down.
Would give us clarity and everyone would be on the same page. I remember the worksheet @ put together for the JoPM paper. We need something like that for this work too. Else, we would be repeatedly meandering and losing track.
[16/07, 12:52]hu4: A new study has just been released from the Stanford ARISE group (Ethan Goh) on arXiv. First DONOHARM. It's the first landmark study on safety of clinical AI.
We have to incorporate learnings from it in our methods. @ @ @ please have a look.
[16/07, 18:12]hu2: Let's aim to publish this project also in arxiv
[16/07, 18:21]hu4: arXiv preprint is a first step, no paper is considered published unless it has passed peer review. Am sure the above Stanford study will appear shortly in a Nature/JAMA/NEJM tier journal
[16/07, 19:52]hu1: I believe this is a very insightful paper, and incredibly useful in our project!
- We cannot catch the "harms of omission" using mcqs alone. This is because, in mcqs, all the safe options are laid out on the screen for the AI to choose from. Phase 2 will point out if AI forgets to mention any critical safety warning, or miss out any red flag.
- Here, they've used LLM as a judge, exactly how @ suggested. The Stanford team scaled the grading using expert defined checklists, which is pretty cool!
- I believe it will be much easier to frame phase 2 as yes/no, instead of making it completely qualitative.
[16/07, 19:54]hu1: I think we should start with phase 1 as of now, and later develop phase 2 as we get an understanding of everything. This will ensure that we keep our stats simple and not complicate things in the beginning itself.
[16/07, 23:14]hu4: Is AMBOSS LiSA a proprietary model? Can we include it in our list of models? The Stanford study says it's the best in patient safety
[16/07, 23:25]hu1: Is this the AI tool used to surf through the AMBOSS library?
[17/07, 00:45]hu4: I think so
[17/07, 08:35]hu1: I'd like to share something just to make sure we are all completely aligned and can lock down our methodology. Here is a quick "gist" of the project and how we are framing it for this first paper:
[17/07, 08:35]hu1: *Study Design and Framework*
This study is designed as a comparative methodological evaluation of large language models (LLMs) using Patient Journey Records (PaJRs) as longitudinal, real world benchmarks of clinical reasoning. Unlike conventional medical AI benchmarks, which largely emphasize textbook diagnostic accuracy and evidence based management, this framework evaluates reasoning across authentic patient journeys occurring within resource-constrained Indian healthcare settings.
The benchmark is developed around longitudinal PaJRs that *document the complete trajectory of patient care, including evolving clinical observations, diagnostic uncertainty, therapeutic decisions, resource limitations, and eventual patient outcomes.* Each PaJR case serves as a real world reasoning pathway rather than a single clinical vignette.
The first benchmark will be illustrated using Case 1, a 50 year old woman with Russell's viper envenomation complicated by severe antivenom induced anaphylactic shock. *This case was selected because it presents a complex sequence of clinical decision making, competing therapeutic considerations, resource constraints, and a documented real-world outcome*, enabling assessment of longitudinal clinical reasoning rather than isolated factual knowledge.
Cases will be selected using a predefined eligibility framework to ensure methodological consistency and adequate complexity for benchmarking clinical reasoning. Eligible Patient Journey Records (PaJRs) are required to represent a longitudinal clinical course spanning hours to months, and contain at least five sequential clinical reasoning checkpoints with multiple evidence decision loops.
Each case will include patient demographics (age and sex), socioeconomic and geographical context, complete vital signs, two to five clinically relevant comorbidities with current medications, and sufficient positive and negative clinical findings to support differential diagnosis. Each case contains 7 multiple choice questions with one correct answer and one to three deliberate distractors, centered around a single primary diagnosis. Finally, all cases will be situated within real Indian healthcare settings and explicitly incorporate resource limitations, such as restricted access to diagnostic investigations, specialist care, or advanced therapeutic interventions, to reflect the complexities of real world clinical decision-making in resource constrained environments.
*The ODAO Framework*
Clinical reasoning will be evaluated using the”Observation–Diagnosis–Action–Outcome” (ODAO) Framework, a structured framework developed to assess reasoning throughout the patient's clinical journey rather than solely evaluating diagnostic accuracy.
Example, for the viper’s case:
Observation (O): Interpretations of the patient's evolving clinical presentation, like laboratory findings & physiological deterioration.
_clinical image_
Diagnosis (D): Identification of the principal clinical problem.
_envenomtion_
Action (A): Selection of the most appropriate management strategy after balancing immediate patient safety, available healthcare resources, and treatment risks.
_avs failed. Waited for dialysis or ventilation required_
Outcome (O): Anticipation of the patient's likely clinical trajectory, possible complications, and long term outcomes based on preceding management decisions.
_recovered apparently from systemic envenomation which was then retrospectively assumed to be mild!_
*The ODAO framework therefore evaluates whether an LLM can maintain coherent clinical reasoning across an evolving patient journey instead of generating isolated correct answers at an “individual timeframe”.*
*Evaluated Models and Systems*
The intervention comprised the Patient Journey Record (PaJR) clinical reasoning system, implemented on Gemini 3.1 Flash Lite. The system utilizes longitudinal Patient Journey Records to support clinical reasoning across the continuum of patient care.
The comparator models will comprise the following: (reasoning enabled) Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro, Gemini 2.5 Pro, Gemini 3.5 Flash, and MedGemma. Each model will receive identical clinical information and standardized prompts to ensure fair comparison.
*No model will be provided with the patient's eventual outcome during the initial evaluation. Responses therefore will reflect prospective clinical reasoning based exclusively on information available at the corresponding stage of the patient journey.*
*The Evaluation Process*
The evaluation will be conducted using a standardized multiple choice question (MCQ) based format on each case. For every clinical case, seven multiple choice questions will be developed to assess predefined domains of clinical reasoning.
The MCQs will be developed by the investigator team based on the documented clinical course and reviewed before evaluation.
Seven MCQs were developed for each case to evaluate patient data understanding (Observation), clinical reasoning and decision support (Decision), action recommendation and care planning (Action), outcome prediction and follow-up (Outcome), temporal reasoning, adaptability to resource-constrained healthcare settings, and communication, safety, and reliability. All evaluated models will receive the same case and identical MCQs under standardized testing conditions. Each question contains a single best answer and one to three carefully constructed distractors. Each correct response will receive one point, yielding a maximum score of seven for each clinical case.
The primary evaluation will be performed without any human intervention, allowing each model to independently generate its responses.
Following the primary assessment, the complete interaction history will be saved. If a model demonstrates a clinically important omission (ignored red flags) or any unsafe recommendation, subsequent human guidance/prompts and the model's revised responses will be documented separately.
*This will be done to distinguish independent correct reasoning from responses that became correct only after expert steering.*
Model responses will independently be reviewed against the predefined reference answers, and compared with the documented clinical trajectory of the corresponding Patient Journey Record. This comparison of model generated reasoning with real world patient outcomes will enable benchmarking of each model's ability to reason prospectively while remaining aligned with real world patient outcomes rather than typical textbook scenarios.
Primary Outcome:
• Autonomous LLM performance (0–7 score)
Secondary Documentation:
• Human interventions
• Safety issues/red flags
• Response revisions
• Alignment with real-world patient trajectory
[17/07, 09:05]hu6: I think ODAO is strong, but the benchmark may also need to evaluate the continuity of reasoning between stages, not only whether a model gets each individual checkpoint right.
A model could score well across seven MCQs while still losing prior facts, dropping unresolved uncertainty or constraints, contradicting itself, failing to preserve corrections, or even arriving at the right answer for the wrong reason.
It may be worth adding a trajectory fidelity layer: does the model preserve a coherent, evolving representation of the patient across time—retaining relevant observations, updating appropriately, carrying corrections forward, and avoiding retrospective rewriting once the outcome is known?
The human-steering data could be especially valuable here: not just whether intervention was required, but what had to be corrected, whether the correction was integrated, whether it persisted, and whether the same failure recurred. That gets at correction persistence, epistemic memory, and recursive failure.
I would also separate outcome concordance from reasoning quality. Recovery does not necessarily validate the pathway that preceded it. The viper case itself seems to show the danger of outcome-based epistemic reversal if later recovery is used to retrospectively redefine the original severity as mild.
So maybe the missing axis is simply this:
ODAO evaluates the content of reasoning across stages. Trajectory fidelity evaluates whether the reasoning itself survives the journey.
[17/07, 09:43]hu4: Our questions for each case should include atleast 1-2 questions which can not be reliably answered from the information present in the PaJR record.
We should also evaluate the LLMs on their ability for refusal to answer. Atleast 1 question per case should be designed for this.
[17/07, 11:19]hu2: Agree
[17/07, 11:19]hu6: I really like this. I’ve learned quite a bit about this from stress-testing PaJR against the 42M record.
I tested whether it could:
* distinguish documented fact, patient report, inference, unresolved question, and unsupported conclusion;
* resist filling evidentiary gaps with plausible-sounding clinical facts;
* preserve corrections after they were explicitly made;
* carry uncertainty forward instead of silently resolving it;
* avoid converting absence of documentation into documentation of absence;
* resist rewriting earlier uncertainty based on later information; and
* maintain those boundaries across repeated iterations of the same longitudinal case.
One of the hardest problems wasn’t simply whether the system knew when it lacked enough information to answer, but whether it could preserve that epistemic boundary over time.
Some AI systems may correctly say, “This cannot be determined from the available evidence.” PaJR, in several of my stress tests against the 42M record, often did something different: it filled the evidentiary gap with a plausible-sounding interpretation and presented it as though it occupied the higher seat of power in the clinical encounter.
Even after correction, the harder test was whether that correction survived. Later iterations could restore an unsupported claim, convert inference into fact, treat absence of documentation as documentation of absence, or lose an evidentiary boundary that had already been explicitly established.
Oddly enough, that began to resemble parts of the patient’s own trajectory: brief encounters with multiple physicians, each working from a siloed perspective and a fragmented record, where provisional or subjective interpretations could acquire the appearance of certainty without the full longitudinal trajectory ever being reconstructed.
So I wonder whether these deliberately unanswerable questions could also be carried forward across later checkpoints. Knowing when not to answer tests epistemic restraint. Remembering what remains unknown—and refusing to rewrite that uncertainty until new evidence actually resolves it—tests longitudinal epistemic integrity.
That may fit directly within trajectory fidelity.
