🔍 Read the full analysis: Microsoft’s AI And Doctors: What The Science Paper Focused On on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A paper published in Science on October 22 reports that Microsoft’s MAI-DxO system scored 83.2% on 304 complex medical cases, compared with 36.1% for generalist physicians using reference materials. The study used historical, text-based cases rather than real patients, and its physician comparison group did not include specialists or teams.
Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 complex medical cases in a paper published October 22 in Science, compared with 36.1% for generalist physicians who could use textbooks and the internet. The study reports a large performance gap on the benchmark, but it tested historical, text-only cases in a simulator, not patients receiving care.
The paper, “Sequential diagnosis with language models,” also reports results on 325 emergency department cases: MAI-DxO scored 80.3%, while physicians scored 46.8%. On the complex-case benchmark, physicians without reference materials scored 22.3%. Microsoft published the results, and the paper appeared in the peer-reviewed journal Science. Peer review evaluates a paper’s methods and reporting; it does not by itself establish that an outside group has reproduced the results.
The benchmark was designed to test how a system gathers information, rather than asking it to select a diagnosis after receiving a complete case file. Each case began with a short summary. The AI or physician could request further details, such as a question or test, from a simulated gatekeeper holding the case information. The benchmark counted both the final diagnosis and the estimated cost of obtaining additional information.
MAI-DxO uses a language model in a panel of virtual physicians that propose diagnoses, request the next detail or test, and challenge one another over multiple rounds. Microsoft’s published chart also shows GPT-5 alone scored 78.9% on the complex cases, without the MAI-DxO orchestration layer. The source account says that layer added about four percentage points and lowered estimated test costs; the study’s reported cost figures are estimates, not observed hospital spending.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
Why the Doctor Comparison Matters
The results are relevant to hospitals and patients because they concern a central clinical task: deciding what information to gather and which diagnosis best fits a case. But the figures should not be read as evidence that MAI-DxO is ready to replace doctors or improve outcomes in routine care. The benchmark measured performance against case records, not whether using the system leads to faster treatment, fewer complications, or better patient health.
The comparison group shapes the headline. The physicians were generalists working alone, while the AI system was organized as a panel of agents. The study did not compare MAI-DxO with specialist teams, even though difficult cases may be referred to specialists or discussed among clinicians in practice. The cases were also selected for complexity, so the results do not show how the system would perform across the full range of patients, including people with common or harmless conditions.
The benchmark’s cost estimates add another point of interest, but they are not a forecast of what a hospital would pay. The source account describes the prices as approximations based on billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge. Actual costs and clinical value would depend on how the system is used and whether its recommendations change care.
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Built
The paper’s benchmark, called the Sequential Diagnosis Benchmark, attempts to represent diagnosis as a series of decisions: ask a question, request a test, consider the result, and decide what to do next. This differs from an exam-style test in which all case details are already supplied. The method is intended to capture both diagnostic accuracy and the cost of information gathering.
The source account notes that some methodological details come from an earlier 2025 preprint, and may not match the final journal version exactly. In that earlier version, a separate model judged diagnoses against the case’s published final diagnosis. Physicians re-graded a sample, with good but imperfect agreement. If a requested result was absent from the case record, the gatekeeper generated a synthetic result. The study also assigned estimated prices to tests using billing codes and a hospital price list.
An earlier, non-peer-reviewed version appeared on arXiv in June 2025 and reported a wider gap, with the system then using OpenAI’s o3 model. The Science paper is the later publication and reports results for the version evaluated there. Microsoft’s blog post highlighted ratios between the system’s scores and physician scores; those ratios describe this benchmark and should not be treated as a general measure of performance in clinical practice.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft CEO of AI Mustafa Suleyman, speaking to TechCrunch
As an affiliate, we earn on qualifying purchases.
Limits Beyond the Case Records
Performance with real patients remains unestablished. The tests described in the source used historical text cases and did not include physical examinations or real-time patient interaction. The study therefore does not answer how the system would handle incomplete or changing information in clinical settings, or whether clinicians would follow its suggestions.
The comparison also leaves open how MAI-DxO would perform against specialists and clinical teams. The source account says the NEJM cases contained no healthy patients and no harmless explanations, limiting what the benchmark could show about false alarms. A system tested only on difficult cases cannot establish how often it might recommend unnecessary testing or identify disease where none is present.
Other limits include the use of a model to score diagnoses, simulated results for tests missing from case files, and estimated prices. The paper’s full text is behind a paywall, and some of the methodological detail available in the source comes from the earlier preprint. Independent replication and evidence from clinical use are not established by the information provided.
As an affiliate, we earn on qualifying purchases.
Prospective Hospital Study Planned
The next development described in the source is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence about how people use the system in a clinical environment, rather than how it performs on archived cases.
The source does not provide a start date, enrollment target, study protocol, or expected results for that work. Until such evidence is reported, the Science paper supports a narrower conclusion: MAI-DxO performed strongly on the specified simulated benchmarks, while its safety, usefulness, and effect on patient outcomes in routine care remain unresolved.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Microsoft’s study find?
The paper reports that MAI-DxO scored 83.2% on 304 complex cases and 80.3% on 325 emergency department cases. Physicians scored 36.1% on the complex cases with reference materials and 46.8% on the emergency department cases.
Were real patients diagnosed by MAI-DxO in this study?
No. The results came from historical case records in a text-based simulator. The study did not test whether use of the system improves care or patient outcomes.
Who were the physicians in the comparison?
The source account describes primary care doctors and hospital generalists working individually. Specialists and physician teams were not included in the comparison group described there.
Does peer review mean the results have been independently confirmed?
No. The paper was published in the peer-reviewed journal Science, but the source account says there has not been an outside reproduction of the results. Peer review is not independent replication.
What is expected to happen next?
A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would use the system alongside their usual care. Details such as timing and study size were not provided in the source.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
