Recent studies conducted by a collaborative team from prestigious institutions—including Harvard, Stanford, the Massachusetts Institute of Technology, and the University of Alberta—reveal that the latest generation of artificial intelligence tools can surpass seasoned physicians when it comes to diagnosing intricate medical conditions. This groundbreaking development underscores the potential transformative impact of AI on the medical field.
“Our study indicates that large language models have outperformed most benchmarks related to clinical reasoning, highlighting an urgent need for prospective trials,” the researchers remark. They advocate for the integration of these models as valuable supplementary diagnostic tools alongside medical professionals in clinical environments.
In a recently published paper in the journal Science, the research team evaluated the efficacy of OpenAI’s o1-preview model against five particularly challenging sets of benchmark cases, which include the New England Journal of Medicine’s clinicopathological case conferences—an authoritative tool for assessing medical acumen for over a century.
Additionally, they harnessed data from contemporary real-world cases in an emergency department at a Boston hospital, seeking AI-generated diagnoses based on information gathered at three critical time points: initial triage, examination by an emergency room physician, and hospital admission. At each stage, both the AI model and human doctors were tasked with providing a “differential diagnosis,” which entails listing the five most likely conditions affecting the patient.
The findings were remarkable: the o1-preview model accurately diagnosed 78.3 percent of historic cases and proposed an appropriate plan for subsequent tests. Regarding the emergency department cases, the model’s differential diagnosis matched exactly or was very close to the correct diagnosis in 67.1 percent of instances during triage, 72.4 percent following the physician’s examination, and a striking 81.6 percent upon hospital or ICU admission. Its performance notably outpaced that of two attending physicians in the initial stages, while it performed comparably to them at the final assessment stage.
“We previously believed that earlier models could revolutionize medicine,” reflects co-author Liam McCoy, a neurology resident at the University of Alberta who contributed during a focused residency research period. “Upon reviewing these results, it became clear that we have reached a pivotal moment. With careful deployment, these models hold incredible promise for enhancing human-machine collaboration.”
A “Thinking” Machine
McCoy emphasizes that the OpenAI o1-preview represents a significant advancement over prior AI models such as ChatGPT. This new model employs a cyclical “thinking” process, allowing it to analyze options and validate its own logic before arriving at a conclusion. Having already shown proficiency in domains like mathematics and software engineering, the researchers were keen to assess its utility in medical settings.
“This model has outperformed human capabilities across a range of tasks,” McCoy shares. “While it’s not ready to fully replace doctors, it presents numerous opportunities to enhance medical quality, which is genuinely exciting.”
We thought earlier models were maybe going to change medicine. The moment I saw these results, I knew for sure this is going to change medicine. We’re already at the point where — if thoughtfully deployed — these models could be very, very useful in human-machine collaboration.
McCoy notes that their recent publication engages in a dialogue with a pivotal article from 1959 published in Science, which first envisioned computers contributing to the medical field. “That study laid the groundwork for how we should conceptualize medical diagnosis,” he states. “Now, 67 years later, we have finally arrived at a stage where the models they envisioned can indeed perform a considerable amount of this reasoning.”
While acknowledging the model’s “superhuman” performance in specific tasks, McCoy highlights the concept of the “jagged frontier,” noting that AI also makes glaring errors and, in some instances, can even recommend harmful actions.
Making Medicine More Human
“We observe that the models do exhibit weaknesses and reasoning constraints,” he explains. “The challenge moving forward lies in enhancing their strengths while minimizing their limitations. Additionally, we must carefully consider whether we are deploying models for tasks they excel at or for those where they struggle.”
Some people are justifiably afraid that using AI might be less humanistic, or alienating, or brought in just as a cost-cutting measure. But I really think there are numerous ways to get creative so we can make medicine more human and more caring with these tools.
Once he completes his residency at the University of Alberta Hospital in the next year and a half, McCoy will join the faculty at Beth Israel Deaconess Medical Center. There, he will partake in testing “collaborative teaming” methods, where doctors work in tandem with AI to determine how second opinions may enhance patient outcomes.
He envisions clinical trials that scrutinize the effectiveness of these AI tools in real-life hospital contexts. “The goal would be to compare outcomes between physicians who have access to this tool and those who do not. We would then evaluate whether this impacts diagnosis accuracy, patient satisfaction, and even mortality rates or delays in appropriate treatment,” he explains.
McCoy points to promising advancements such as a recently approved AI tool in the United Kingdom for faster stroke diagnosis and the University of Alberta’s “Jenkins” AI scribe tool, currently undergoing trials across Alberta.
He perceives immense potential for AI across nearly every medical specialty, including training for students (by synthesizing study materials and identifying research) and enhancing palliative care (imagine an AI chatbot ready to answer questions about diagnoses at any hour). Furthermore, AI could assist in communication disorders by interpreting and completing speech for patients experiencing aphasia.
McCoy foresees a future wherein, once these tools are validated through prospective trials and safely integrated into clinical workflows, it may become standard practice to utilize AI, similar to how other advanced technologies like MRIs are now fundamentals of patient care.
“Some people express valid concerns that implementing AI could lead to a less humanistic approach to healthcare, or that it may be used merely as a cost-cutting strategy,” he reflects. “However, I truly believe that there are countless innovative ways to harness these tools to foster a more human and compassionate medical environment.”
