Read the full analysis: Microsoft Says AI Beats Doctors. Here’s What Researchers Assessed on ThorstenMeyerAI.com
TL;DR
Microsoft’s MAI-DxO system scored 83.2% on 304 difficult New England Journal of Medicine cases, compared with 36.1% for generalist physicians using reference materials. The peer-reviewed study tested historical, text-only cases in a simulator—not diagnosis or treatment of real patients—and the results have not been independently reproduced.
Microsoft’s MAI-DxO diagnostic system scored higher than generalist physicians on two sets of challenging medical cases in a study published October 22 in the peer-reviewed journal Science. The findings show performance on historical, text-only cases in a simulated setting; they do not establish that the system outperforms doctors treating patients in hospitals or clinics.
On 304 complex case records from the New England Journal of Medicine (NEJM), Microsoft reported 83.2% diagnostic accuracy for MAI-DxO. The physician comparison group scored 36.1% when allowed to use textbooks and the internet, and 22.3% without those resources. On a separate set of 325 emergency department cases, the reported scores were 80.3% for the AI system and 46.8% for physicians.
The benchmark began with a short case summary. Participants had to request additional information, such as a question or test result, from a simulated case file. The system was scored on its final diagnosis and the estimated cost of obtaining information. This design aimed to test how a diagnosis is reached, not simply whether a model can choose the right answer when given a complete case.
MAI-DxO uses a language model in a panel of virtual physicians that propose diagnoses, choose follow-up questions or tests and evaluate one another’s suggestions. Microsoft’s chart also shows that GPT-5 alone scored 78.9% on the complex cases. That comparison suggests much of the reported diagnostic score came from the underlying model, while the orchestration system contributed a smaller additional gain and reduced estimated testing costs, according to the source account.
Science · 22 Oct 2026
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
MAI-DxOno test budget83.2%GPT-5 alonesame benchmark, no orchestration78.9%Physicians, open booktextbooks and internet36.1%Physicians, closed bookno references22.3%
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
Generalists only: primary care doctors and hospital generalists. No specialists, no teams.Working alone: no colleagues; in the preprint also no search engines or language models.Text-only, historical cases: no real patients, no physical exam, no imaging.Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Mean for Care
The results matter because diagnostic tools could affect which tests are ordered, how quickly conditions are identified and how much care costs. A system that asks useful follow-up questions while limiting unnecessary testing could eventually support clinicians, particularly when cases are difficult. But the study measured performance under a controlled benchmark, not changes in patient outcomes, safety, waiting times or real-world spending.
The gap between the AI and physician scores is substantial within the study’s setup, but the comparison was not between MAI-DxO and specialist teams. The doctors were generalists working individually, while the AI system brought together multiple virtual agents. Difficult teaching cases may also be routed to specialists in actual care. The scores therefore do not establish that the AI is better than the full clinical process patients encounter.
For hospitals, insurers and patients, the practical takeaway is that the paper presents a promising benchmark result, not evidence that MAI-DxO is ready to replace physicians or independently direct care. Adoption would require evidence from patient-facing settings, including how often the system misses dangerous conditions, triggers unnecessary tests or changes clinicians’ decisions.
How the Benchmark Was Built
The Science paper is titled Sequential diagnosis with language models. The source account says it was published October 22, 2026, after being received in April and accepted in August. All 15 listed authors had Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman. An earlier version appeared as a non-peer-reviewed preprint in June 2025 and reported a different comparison, using an earlier model. The current figures cited here are the ones attributed to the Science publication.
The benchmark’s design reflects one part of clinical reasoning: doctors often begin with incomplete information and decide what to ask or test next. In the study, every additional detail had to be requested from a gatekeeper model holding the case file. The source account says missing test results could be generated synthetically, and estimated test prices were based on billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge. These are simulated estimates, not observed costs from treating patients.
The physician comparison also has limits. According to the source account’s description of the earlier study materials, the clinicians were primary-care doctors and hospital generalists, not specialists; they worked alone, and the cases were difficult published examples. The Science study added an open-book comparison, but the overall evaluation remained a case-record benchmark, rather than a trial in which doctors and AI jointly cared for patients.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft CEO Mustafa Suleyman, speaking to TechCrunch
Limits Before Patient Use
The reported accuracy figures do not show how MAI-DxO performs with real patients, physical examinations, imaging or live clinical workflows. The source account says a model evaluated diagnoses against published answers, and physicians’ re-grading of a sample showed good but imperfect agreement. It also reports that the complex NEJM set did not include healthy patients or harmless explanations, meaning the benchmark could not measure false alarms in the usual way.
Microsoft’s results have appeared in a peer-reviewed journal, but peer review is not independent replication. The study’s authors are all listed as Microsoft-affiliated, and the source material does not identify an external reproduction of the findings. It is also unclear from the provided material how the final journal version handles every method detail described in the earlier preprint; those details may have changed.
The results do not establish whether the system would improve outcomes, reduce total health-care costs or perform reliably across patient populations and institutions. Nor do they show how clinicians would respond when the AI’s recommendation conflicts with their judgment. Those questions remain open pending prospective evaluation.
A Prospective Hospital Study
The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients are expected to interact with the system alongside their usual care. That setting could test performance with current clinical information and reveal how the tool fits into actual care, though the provided material does not specify the study’s start date, size, endpoints or expected publication date.
Readers should look for whether the evaluation measures more than diagnostic accuracy: patient safety, missed and false diagnoses, unnecessary tests, clinician workload and costs are all relevant to judging clinical usefulness. Until those results are available, the Science paper supports a narrower conclusion: MAI-DxO performed strongly on the cases and under the conditions tested, while its value in routine patient care remains unproven.
Key Questions
Did Microsoft show that its AI is better than doctors?
It reported higher diagnostic accuracy than generalist physicians working alone on selected historical cases. The study did not compare MAI-DxO with specialist teams treating patients, so it does not establish that the AI is better than doctors across clinical care.
What cases did MAI-DxO assess?
The study used 304 complex NEJM case records and a separate set of 325 emergency department cases. Participants began with brief summaries and requested additional details in a simulated setting.
Was the system tested on real patients?
Not in the benchmark described. The cases were historical and text-based, with no real-time patient interaction, physical examination or imaging. A prospective study at Beth Israel Deaconess Medical Center is reported as the next step.
Does peer review confirm the results independently?
No. Publication in Science means the paper underwent journal peer review, but that is not the same as independent replication. The source account says all listed authors had Microsoft affiliations and does not report an outside reproduction.
Source: ThorstenMeyerAI.com