The Paper Behind Microsoft’s Claim That AI Beats Doctors

🔍 Read the full analysis: The Paper Behind Microsoft’s Claim That AI Beats Doctors on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get hardware and tech essentials delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft published a peer-reviewed study of MAI-DxO in Science on October 22, reporting that the diagnostic AI system reached 83.2% accuracy on 304 difficult medical cases. The comparison was against individual generalist physicians working through historical case records in a text-only simulator—not against specialists or medical teams treating patients.

The paper, titled Sequential diagnosis with language models, reports that generalist physicians scored 36.1% with textbooks and internet access and 22.3% without those resources on the 304 complex cases. In a separate set of 325 emergency department cases, Microsoft reported 80.3% accuracy for MAI-DxO and 46.8% for physicians. The figures are the study authors’ results; the system was not tested on real patients in this evaluation.

The benchmark, called the Sequential Diagnosis Benchmark, gives each participant a short case summary. The participant then requests additional information, such as a test result, from a simulated case-file holder. Tests carry estimated costs, so the evaluation considers both the final diagnosis and the requests made along the way. MAI-DxO uses a language model in a process that simulates a panel of virtual physicians proposing diagnoses, asking for information and challenging one another.

Microsoft’s published results also show GPT-5 alone scored 78.9% on the complex cases, compared with 83.2% for MAI-DxO. That suggests most of the reported accuracy came from the underlying model, with the orchestration system adding a smaller gain. Microsoft also reports reduced estimated test costs from its system; the supplied material does not provide enough detail to establish how those estimates would translate to clinical practice.

What the Scores Mean for Care

The study addresses a real challenge in medical AI evaluation: diagnosis often involves deciding what to ask or test next, not simply selecting an answer after receiving a complete case file. A benchmark that includes information requests and estimated test costs examines more of that process than a conventional exam-style question set.

But a high score in this setting is not evidence that the system is more effective than doctors in routine care. The cases were historical and text-based, without physical examinations or real-world imaging review. The comparison was also between a coordinated AI process and solo generalists, while difficult cases may be handled by specialists or teams in practice. Readers, hospitals and insurers should treat the reported figures as benchmark performance—not proof of improved patient outcomes, safer decisions or lower costs in clinics.

The distinction matters because clinical adoption can affect decisions about testing, referrals and treatment. False alarms and missed diagnoses carry consequences that an accuracy score alone cannot capture. The study can inform further research, but it does not establish that patients should rely on MAI-DxO instead of a clinician.

How Microsoft Built the Comparison

The Science paper was published October 22, 2026, after being received in April and accepted in August, according to the source material. Its 15 authors are listed with Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman. An earlier version appeared on arXiv in June 2025; the source says that preprint reported an 80% AI score against 20% for doctors, using an earlier model setup. Those earlier figures should not be conflated with the results in the published paper.

Some methodological details in the supplied account come from the earlier preprint, rather than the full Science article, which it says is behind a paywall. In that version, a separate model judged diagnoses against the recorded final diagnosis, and physicians reviewing a sample showed good but imperfect agreement with those scores. When a requested test result was absent from the case record, a gatekeeper model generated a synthetic result. The cost estimates used billing codes and a 2023 U.S. hospital price list, alongside a fixed $300 physician-visit cost; the authors described the figures as first-order approximations.

The preprint’s physician group included 21 U.S. and U.K. generalists: 17 primary-care doctors and four hospital generalists, with a median of 12 years’ experience. In that version, doctors were assessed on 56 held-out cases, while the AI was assessed on all 304. The published Science paper adds an open-book physician comparison, but the supplied source does not detail every change between the preprint and final study.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft chief executive Mustafa Suleyman, speaking to TechCrunch

Limits the Benchmark Leaves Open

The study does not show how MAI-DxO performs with live patients, physical examinations, imaging or a full clinical record. Nor does the supplied material report whether the system improves patient outcomes, reduces diagnostic delays or changes the rate of harmful errors. Peer review evaluates a paper’s methods and reporting; it does not mean that independent researchers have reproduced the results.

The case selection also limits what the accuracy figures can say about everyday care. The source says the NEJM cases were selected for complexity and contained no healthy patients or harmless explanations. In the preprint, the authors noted that this meant the study could not measure false alarms. It is also unclear from the supplied account how closely the final paper’s physician protocol and scoring procedures match those in the preprint.

Finally, the cost results rely on estimated test prices rather than observed spending in hospitals. The published scores cannot establish whether the system would save money after accounting for clinician review, follow-up testing, software costs or errors. Those questions remain open pending evaluation in clinical settings.

Testing MAI-DxO Alongside Patients

The next identified step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients are expected to interact with the system alongside their normal care. That design could provide evidence about how people use the tool in a clinical setting, but the supplied account does not give a start date, enrollment target or planned outcome measures.

Results from that study will be needed to assess performance beyond curated case records, including whether the system’s recommendations help clinicians and how often they prompt unnecessary or missed tests. Until those findings are available, Microsoft’s 83.2% result remains a score on a defined benchmark, not a measure of how well the system diagnoses patients in hospitals.

Key Questions

What did Microsoft’s study find?

Microsoft reported that MAI-DxO scored 83.2% on 304 difficult historical cases. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without them in the published comparison described by the source.

Was MAI-DxO tested on patients?

No. The reported benchmark used historical, text-based case records in a simulator. The source says a separate prospective study alongside normal care is planned at Beth Israel Deaconess Medical Center.

Were the doctors compared with specialists?

The comparison was with generalist physicians working individually, not specialists or clinical teams. The source says difficult cases like those in the benchmark may be handled by specialists or teams in real practice.

Does peer review mean the results were independently confirmed?

No. The paper’s publication in Science means it underwent peer review, but the supplied source says independent replication has not been established. Peer review is not the same as reproducing a result in a separate study.

What is the next step for the system?

The source identifies a planned prospective study at Beth Israel Deaconess Medical Center, where consenting patients would interact with the system alongside their usual care. Its timing and outcome measures were not provided.

Source: ThorstenMeyerAI.com