Performance of a large language model on the reasoning tasks of a physician

The Harvard and Beth Israel team that has spent two years testing whether chatbots can out-reason doctors finally landed the capstone in Science: across six experiments, OpenAI's o1-preview matched or beat attending physicians at diagnostic reasoning, including on 76 real emergency-room cases pulled from raw hospital records rather than tidy textbook vignettes.

The preprint was titled 'Superhuman performance of a large language model on the reasoning tasks of a physician.' Peer review cut it to 'Performance.'

The headline result holds up. Given only what a nurse knows at the triage desk — the information-poorest moment in a visit — the model put the right diagnosis on the table 67% of the time versus 55% and 50% for the two human doctors. It was strongest exactly where the humans had the least to go on.

But the team's December 2024 preprint had called this 'superhuman performance.' The peer-reviewed version is titled, simply, 'Performance.' The walk-back is in the data: the model's edge clusters at cheap-to-be-wrong moments — early triage, rare puzzle cases — and evaporates on the 'cannot-miss' calls, the life-threatening diagnoses where being right is the whole job. There it beat neither ChatGPT nor the doctors. The real-ER test was also a retrospective chart review; the model never managed a live patient, and its human comparators were internal-medicine doctors, not ER specialists.

So the finding that survived review is narrower and more interesting than the one that didn't: a reasoning model is most useful as a second opinion precisely when a clinician is flying blind, and least trustworthy on the emergencies where a confident wrong answer does the most harm.