Microsoft's AI Diagnostic: Outperforming Doctors in Medical Simulations

Microsoft has released a study claiming its AI diagnostic system outperforms human physicians in complex cases. The study, published Thursday in Science, found that the system, MAI-DxO, correctly identified conditions more than twice as often as generalist doctors in a simulated setting. The company's head of AI, Mustafa Suleyman, called the result a significant step toward "medical superintelligence." The system works by coordinating a panel of AI agents that can question a patient, order tests, and debate a potential diagnosis.

In the study, the system achieved an 83% accuracy rate on complex cases from the New England Journal of Medicine. By comparison, primary care doctors and general internists working individually with reference materials were correct 36% of the time. It is important to note that cases of this complexity are typically handled by specialists or physician teams, who were not part of this comparison. The system's performance varied when used with different underlying AI models. Its best results came when powered by OpenAI's GPT-5 and GPT-5.5 models. The study's data also suggests that much of the performance improvement comes from the more powerful underlying AI models, with Microsoft's orchestration system adding further gains on top.

Suleyman, who leads Microsoft's AI unit, stated the company is working with the Mayo Clinic to train a foundation model for healthcare, with a release planned for next year. He believes this model will enable the company to "make really enormous leaps here" in medical care. Suleyman predicts that by 2027, the company will be on the verge of a breakthrough for medical records. He anticipates the ability to stream highly accurate predictions about how a medical record might unfold, adding that such foresight for a clinician is gold dust. Suleyman has spent his career trying to use AI to solve medical issues, previously co-founding DeepMind, which was acquired by Google in 2014.

While the use of generative AI in healthcare is still early, Suleyman expects healthcare models to get an order of magnitude better in capability within the next 24 to 36 months. He believes that once a flywheel effect is established, progress happens very quickly. Microsoft is seeking real-world evaluation of its system through a new partnership with Beth Israel Deaconess Medical Center in Boston. The partnership will evaluate the clinical reasoning capabilities of DxO across different hospital settings. The study will involve patients receiving standard treatment while also engaging with DxO. Microsoft and its partners will then compare the system's recommendations against those of human doctors. Suleyman expressed commitment to developing state-of-the-art medical clinical decision support products over the next few years, stating that this focus could differentiate Microsoft from other AI developers.