New Delhi: Artificial intelligence (AI) continues to make rapid strides, and its growing role in healthcare has now been highlighted by a major new study that compared the performance of AI systems with human physicians in clinical reasoning tasks.
The study, conducted by researchers from Harvard Medical School and Beth Israel Deaconess Medical Center, found that at least one large language model (LLM) demonstrated accuracy comparable to—or in some cases better than—human doctors across a wide range of medical decision-making scenarios.
The research aimed to evaluate whether AI could perform tasks that physicians routinely handle, including diagnosing illnesses, making emergency room decisions, and determining the next steps in patient management.
According to the findings, the AI model performed strongly across multiple benchmarks. In several cases, it matched or exceeded the diagnostic accuracy of experienced physicians, particularly in early-stage clinical decision-making.
Co-senior author Arjun Manrai noted that the model outperformed both earlier AI systems and physician baselines in many tests, signalling a significant leap in capability.
One of the key components of the study involved testing AI performance in emergency room settings. Researchers used 76 real cases from Beth Israel Deaconess Medical Center, where decisions ranged from prioritising care to determining whether patients required admission to intensive care units.
The study compared diagnoses generated by two attending physicians with those produced by AI models developed by OpenAI, specifically the o1 and 4o models.
To ensure fairness, another set of physicians—unaware of the source of the diagnoses—evaluated the results. The findings revealed that the AI model o1 performed either on par with or slightly better than human doctors at various stages of diagnosis.
Importantly, the AI was provided only with the same information available to doctors at each stage, drawn directly from electronic medical records, without any additional processing.
The outcomes surprised even the research team. Co-senior author Adam Rodman admitted that initial expectations were modest, but the model's performance exceeded them.
At early decision points in emergency care—often the most critical stage—the AI consistently demonstrated strong diagnostic capabilities, matching or surpassing physicians in accuracy.
These findings suggest that AI tools could potentially support clinicians in high-pressure environments where rapid and accurate decisions are essential.
Despite the promising results, researchers cautioned against interpreting the findings as a sign that AI can replace doctors. The study emphasised that AI systems are not yet ready for independent clinical use.
Experts pointed out that while AI may correctly identify a diagnosis, it could also recommend unnecessary tests or interventions that may pose risks to patients.
Human oversight remains essential, particularly in ensuring patient safety and making nuanced decisions that go beyond data analysis.
The study highlights the potential for AI to be used as a supportive tool in medicine rather than a substitute for human expertise. Researchers believe that such systems could assist doctors by improving diagnostic accuracy, reducing workload, and enhancing decision-making.
However, they stressed the need for rigorous clinical trials and real-world testing before AI can be widely adopted in healthcare settings.
The findings mark a significant milestone in the intersection of AI and medicine, demonstrating that advanced models are capable of handling complex clinical reasoning tasks. While the technology shows promise, its role is likely to remain complementary, supporting rather than replacing healthcare professionals.
As research continues, the focus will be on ensuring that AI systems are safe, reliable, and integrated responsibly into medical practice.