To the Editor—Brodeur et al.^[1] demonstrate that large language models achieve impressive accuracy on structured clinical vignettes, surpassing physician benchmarks in controlled settings. This is a genuine technical advance. Yet these tasks are, in essence, static medical examinations—closed questions with preset answers at a single time point, no different in principle from having an AI sit a licensing exam. 

Real clinical practice is an open, iterative process: provisional reasoning, targeted investigation, empirical treatment, and serial revision as new data emerge. A core competence of physicians is the ability to detect errors through subsequent patient outcomes, correct the diagnosis promptly, and learn from the experience. The study offers AI no such mechanism: even if the model errs, it cannot detect the mistake, receive the true outcome, or recalibrate. 

This static architecture creates a dangerous accountability vacuum in real-world deployment. When an AI makes a diagnostic error in live practice, the system itself does not bear adverse consequences or "feel" the cost of its mistake. Machines may repeat errors without assuming liability, while physicians bear ultimate legal and ethical responsibility.^[2] 

Moreover, diagnosis is not merely a cognitive exercise. It also depends on the physician's empathy for the patient, embodied physical examination, tacit contextual judgment, and the human relationship that elicits patient narratives and non-verbal cues. The authors acknowledge excluding physical examination and nonverbal communication,^[1] yet omit that trust itself is part of therapeutics. An AI that answers questions perfectly but has never stood at the bedside cannot replicate this dimension of care. 

In summary, AI excel at structured knowledge—drug interaction alerts, literature retrieval, and initial image triage—but physicians remain irreplaceable for embodied examination, tacit reasoning, situational judgment, and empathic connection.^[3] The future of medicine is physician-led human–machine collaboration, not replace physicians.

References

[1] Brodeur P G, Buckley T A, Kanjee Z, et al. Performance of a large language model on the reasoning tasks of a physician[J]. Science, 2026,392(6797):524-527.

[2] Cestonaro C, Delicati A, Marcante B, et al. Defining medical liability when artificial intelligence is applied on diagnostic algorithms: a systematic review[J]. Front Med (Lausanne), 2023,10:1305756.

[3] Sokol K, Fackler J, Vogt J E. Artificial intelligence should genuinely support clinical reasoning and decision making to bridge the translational gap[J]. NPJ Digit Med, 2025,8(1):345.

The authors have declared that no competing interest exists.

Brodeur and colleagues report that an advanced large language model has eclipsed most benchmarks of clinical reasoning (1). The paper documents real progress, and the emergency department experiment is well-engineered. The authors themselves note in their Discussion that existing benchmarks "may therefore overstate the performance of AI models when using 'messy' data available in more realistic clinical workflows." Hopkins and Cornelisse note in the accompanying Perspective (2) that "passing examinations is not the same as being a doctor, and demonstrating physician-level performance on authentic clinical tasks is a fundamentally harder challenge."

We extend these acknowledgments with three specific concerns about how the most-cited number from the study (89% versus 34% on the Grey Matters management cases) is being interpreted as evidence that AI surpasses physicians in clinical reasoning. The first concern is methodological. The second comes from the paper's own strongest experiment. The third addresses study-design balance.

First, the scoring rubrics share a structural feature that systematically advantages verbose, comprehensive answers. The Bond Score awards full credit for inclusion of the correct diagnosis anywhere in the differential, with the prompt explicitly stating no limit on differential length (Supplementary Text 1A and 1B). On the 101-case overlap with a prior physician study, o1-preview's top-1 accuracy was 66.3% but its top-10 accuracy was 81.2% (Table S2), a 15-point gap that reflects enumeration value rather than diagnostic precision. The Grey Matters management rubric is largely additive across line items: each item earns a small fixed number of points if included, and no item subtracts from the total. Question 1 awards 19 points across 22 line items for the workup of involuntary weight loss, with no penalty for ordering excess or wrong tests (Supplementary Text 3B). CT of abdomen and pelvis receives the same 1 point as screening for food insecurity. (Question 3, on antibiotic management, is a noted exception with explicit 0-point options for clinically wrong management, but the additive pattern dominates the other questions in the example rubric.)

Table S5 case 3-2022 illustrates the resulting scoring problem: an o1-preview test plan listing more than 30 tests across 8 categories (including bone marrow biopsy, four specialty consults, leptospira microscopic agglutination, Ehrlichia PCR, and HLA-B27 typing) for a case ultimately diagnosed as Crohn disease scored 1 ("helpful") on the rubric, which is just one step down from the full score of 2 (“exactly right”). The excessive bone marrow biopsy and four specialty consults piled on top did not drop the score below "helpful." The rubric distinguishes "helpful" from "exactly right" (matching the actual case plan), but provides no mechanism to penalize the inappropriate items added alongside reasonable ones. o1-preview generates substantially longer outputs than the focused physician answers used as comparators. A focused physician answer cannot reach as many rubric items as comprehensive enumeration, even when the reasoning behind it is sharper.

The pattern across the paper's six experiments is consistent with the rubric-bias hypothesis. Where the metric measures focused clinical judgment, AI's advantage is small or absent: cannot-miss diagnoses (Figure 3B, where o1-preview was not significantly higher than physicians, GPT-4, or residents), Landmark diagnostic reasoning (Figure 4B, where o1-preview was not significantly different from physicians with conventional resources, p=0.055, or with GPT-4 access, p=0.076; the Landmark cases are also the only experiment whose cases were never publicly released, specifically to prevent memorization), and probabilistic reasoning (Table S6, where all groups overestimate the likelihood of low-prevalence conditions). Where the metric rewards comprehensive enumeration most heavily (the Grey Matters additive checklist), AI's advantage is largest. The cross-experiment gradient is itself evidence that the rubric type is doing more work than the abstract framing acknowledges.

Second, the paper's own strongest experiment shows an information gradient that is more informative than the headline number. The 76-case ED study (Figure 5) tested o1 against two attending physicians on real emergency department patients at three points in their evaluation: triage (chief complaint, vitals, and nursing note only), ED evaluation (after the initial physician encounter and labs), and admission (full workup). At triage, o1 placed the correct diagnosis at the top of the differential 67.1% of the time, compared with 55.3% and 50.0% for the two physicians, a 12 to 17 percentage point gap. By admission, the rates were 81.6% (o1), 78.9% (physician 1), and 69.7% (physician 2). The difference between o1 and physician 1 at admission is no longer statistically significant.

Same patients, same model, same physicians. The only thing that changes across the three touchpoints is the amount of clinical information available. The pattern shows AI's advantage concentrating where information is sparse and shrinking as information accumulates. This is the strongest piece of work in the paper, and it tells a different story than the headline. Even though AI shows advantage at triage, in real ED practice, no clinical decision turns on the triage differential alone; the next steps (orders, examination, response to therapy) are exactly where the information gradient erodes the AI's edge. The within-experiment gradient is also consistent with the cross-experiment pattern noted above: rubrics that reward enumeration favor AI most when there is little focused information to constrain the differential.

Third, five of the six experiments use historical physician comparators rather than concurrent ones. The Grey Matters numbers come from a 2025 publication (3); the Landmark cases from a 2024 publication (4); the NEJM Healer comparison from a 2024 publication (5); and the probabilistic reasoning vignettes from a 2021 publication (6). The Brodeur runs spanned September 2024 to August 2025, with reasoning effort set to high and the maximum 65,536 token output budget. Historical-control comparisons in any field carry confounders that head-to-head designs eliminate: era effects, scorer drift between graders, and selection in the original cohort. Whatever the magnitude of the apparent advantage, it cannot be cleanly attributed to model superiority over physicians. The 55-percentage-point gap on Grey Matters is the most striking example.

These three observations do not invalidate the paper. They locate where its conclusions are most defensible (the ED experiment, with concurrent comparators and blinded scoring, examined across the information gradient) and where they are most strained (the historical-control comparisons that are now circulating as evidence of AI superiority over physicians). The paper itself is more cautious than the secondary coverage suggests: the limitations section acknowledges that performance gains were not robust on the cannot-miss diagnoses or the Landmark cases.

As the field considers the implications for clinical practice, magnitude claims should be matched to the strongest version of the evidence. On contemporaneous head-to-head testing in real ED data, o1 was narrowly better than two attending physicians where information was sparse and equivalent at admission when full information was available. That is meaningful and worth confirming in larger trials. It is not the gap implied by the 89%-versus-34% framing now in circulation, and it is not what the paper's own internal evidence supports.

Sincerely,

Allen Li, MD

References

1. Brodeur PG, Buckley TA, Kanjee Z, et al. Performance of a large language model on the reasoning tasks of a physician. Science 392, 524 (2026).

2. Hopkins AM, Cornelisse E. AI can reason like a physician—what comes next? Science 392, 466-467 (2026). doi: 10.1126/science.aeg8766

3. Goh E, Gallo RJ, Strong E, et al. GPT-4 assistance for improvement of physician performance on patient care tasks: A randomized controlled trial. Nat Med 31, 1233-1238 (2025).

4. Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: A randomized clinical trial. JAMA Netw Open 7, e2440969 (2024).

5. Cabral S, Restrepo D, Kanjee Z, et al. Clinical reasoning of a generative artificial intelligence model compared with physicians. JAMA Intern Med 184, 581-583 (2024).

6. Morgan DJ, Pineles L, Owczarzak J, et al. Accuracy of practitioner estimates of probability of diagnosis before and after testing. JAMA Intern Med 181, 747-755 (2021).

Brodeur et al. (1) report that OpenAI's o1-preview meets or exceeds physician baselines across six experiments in clinical reasoning. The methodology is careful and the blinding in the emergency department arm is robust. I want to raise a concern that is not about methods but about what the comparator measures, and what it no longer measures, in 2026.

The headline framing — that an LLM outperforms physicians on curated clinical cases — was a meaningful finding when the underlying work was first posted as a preprint in December 2024 (then titled "Superhuman performance of a large language model on the reasoning tasks of a physician"). It is a different finding in 2026. According to the AMA's 2026 Augmented Intelligence Research survey, 81% of US physicians now report using AI professionally, up from 38% in 2023 and 66% in 2024 (2). Purpose-built clinical tools (OpenEvidence, ChatGPT Health, ambient scribes) are widely deployed. The unaided physician described in the paper's comparisons is increasingly a hypothetical reference point rather than a description of contemporary practice.

This matters because the most interesting result in the paper is one the authors report but do not engage with. On the Grey Matters management cases, physicians using GPT-4 scored a median of 41% (IQR 31–54), GPT-4 alone scored 42% (IQR 33–52), and physicians using only conventional resources scored 34% (IQR 23–48). On the landmark diagnostic cases, physicians with GPT-4 scored 76%, physicians with conventional resources 74%, and GPT-4 alone 92%. The model alone outperformed the model-plus-physician dyad. Physicians given the tool barely outperformed physicians without it.

Either this 2024 finding still generalizes, in which case the question of why the joint system underperforms the model is the most pressing one the paper raises and deserves more than a sentence. Or it no longer generalizes, in which case the paper's headline comparison is itself dated, because the relevant comparator is no longer the unaided physician. Either reading points to the same gap: what we need to know in 2026 is what happens between physicians and these tools in real workflows, not whether the model can answer a curated case better than a physician working without one. The accompanying Perspective by Hopkins and Cornelisse (3) recognizes this, citing the same effect from Goh et al. (4) and arguing that "determining the optimal implementation will likely require an evaluation that compares AI alone, clinician alone, and clinician with AI."

A related concern: vignette-based benchmarking, including the CPC, Healer, and landmark cases used here, presupposes that the curation step has already been completed. A real patient has been distilled into a structured presentation in which the relevant labs are surfaced, the timeline is clean, and the chief complaint has been disambiguated from the noise. Much of a clinician's work is upstream of this point. The emergency department arm is the most honest part of the paper precisely because it attempts unstructured data, but even there, model and physician receive the same pre-extracted information at three predefined touchpoints. The authors note this limitation in the discussion. It deserves more weight than a footnote.

None of this is a criticism of the work as performed. It is an observation that the question the paper answers — can a frontier model match physicians on curated cases — has largely been settled in the affirmative across multiple studies (5), and that the field's open questions have shifted to the joint human-AI system, the failure modes of that system, and the curation step that vignette-based evaluation cannot capture. A reflection from these authors, who have done foundational work in this area, on where the field stands now would have been at least as valuable as a further demonstration of model-versus-physician performance on benchmarks the field is already confident the model can pass.

A longer version of this argument is available at https://sparsethought.com/2026/05/03/science-paper/

Gal Sapir, MD, PhD

References

1. P. G. Brodeur et al., Science 392, 524 (2026).
2. American Medical Association, 2026 Augmented Intelligence Research Survey. https://www.ama-assn.org/practice-management/digital-health/more-80-physicians-use-ai-professionally-ama-survey
3. A. M. Hopkins, E. Cornelisse, Science 392, 466 (2026).
4. E. Goh et al., Nat. Med. 31, 1233 (2025).
5. D. McDuff et al., Nature 642, 451 (2025).

Brodeur and colleagues report strong performance of a 2025‑era large language model (LLM) across several diagnostic and management tasks. The results are notable, but the interpretation that these systems have “eclipsed most benchmarks of clinical reasoning” requires a more tempered reading once the study’s methodological and structural context is considered.

The public‑facing risk of this work is not hypothetical. Studies of clinical AI consistently show that patients over‑rely on systems they perceive as “more accurate,” even when those systems provide unsafe or inappropriate advice. Recent Nature evaluations have demonstrated that current LLMs frequently generate incorrect or clinically dangerous recommendations, particularly when confronted with uncertainty, atypical presentations, or missing data. Against this backdrop, media amplification of claims that AI “outperforms doctors” risks eroding the therapeutic alliance that underpins safe care. If vulnerable patients begin to privilege software outputs over clinician guidance, especially in domains such as medication adherence, chronic disease management, or risk communication, the consequences could be profound. The authors themselves acknowledge that their model was tested only on curated, text‑based inputs and that real‑world clinical encounters are “awash with nontext inputs,” yet headlines rarely carry such nuance. High‑profile publications therefore have a responsibility to avoid inadvertently encouraging patient over‑reliance on systems that have not been shown to be safe for autonomous use.

First, the evaluation relies on curated, text‑only inputs. As the authors acknowledge, “Existing benchmarks… rely on the careful work of clinicians to curate and ‘clean up’ cases… which may therefore overstate the performance of AI models when using ‘messy’ data available in more realistic clinical workflows.” Emergency departments are defined by incomplete histories, contradictory documentation, multimodal cues, and time pressure, none of which were represented in the study design. The model’s performance therefore reflects linguistic reasoning under idealized conditions, not the full complexity of clinical judgment.

Second, even within these controlled settings, the results are mixed. The model did not outperform physicians on “cannot‑miss” diagnoses, and its advantage in landmark diagnostic cases was not statistically significant. Probabilistic reasoning gains were modest and case‑dependent. In the emergency department experiment, its most striking finding, the authors themselves describe the task as “best thought of as a proof of concept.” The model generated second‑opinion differentials from text; it did not triage, communicate risk, integrate visual or behavioural cues, or assume responsibility for decisions.

Finally, the study emerges from a highly interconnected research ecosystem with substantial alignment to the commercial AI landscape. The funding portfolio includes the Harvard Dean’s Innovation Award for Artificial Intelligence, the Moore Foundation’s AI‑for‑diagnosis initiatives, and multiple NIH grants supporting computational medicine. Senior authors hold direct industry affiliations, “A.R. is a Visiting Researcher at Google DeepMind” and “E.H. is employed by Microsoft”, while others lead NEJM Healer, a commercial reasoning platform. Although several authors report no personal conflicts, the structural alignment of academic, editorial, philanthropic, and industry interests underscores the need for interpretive caution when claims of surpassing human reasoning are advanced.

Taken together, the study demonstrates that LLMs can outperform physicians on structured, text‑based reasoning tasks. It does not show that they exceed clinicians in real‑world diagnostic practice, nor that they can safely operate in the noisy, multimodal, time‑sensitive environment of modern hospitals. Prospective trials in authentic clinical settings remain essential before drawing conclusions about the role of AI in clinical reasoning.

The New York Times reports that the Florida Atty General has opened a criminal investigation into ChatGPT's role in a campus shooting last year because ChatGPT answered questions the shooter asked in planning and carrying out the murders, including " how do I turn off the aafety on my gun?" just before opening fire. A human who answered these questions would had already been charged as an accessory, authorities say. 

It will be fascinting to see if ChatGPT, Anthropic etc will share any of the medical malpractice risk of their LLM ( or if the LLM is legally responsible for malpractice) if their LLMs become involved in dignosis and treatment decisions. Will clinicans be able to use " Claude told me to do this" as a defense?  Medical textbooks are not liable by providing information used in clinical care, but this is a very different situation, isn't it? 

Will patients be told of their invovlvement? Now while it is expected patients be told AI is helping write notes,  my experience is few are so informed.