An AI Tool Grounded in Evidence-Based Medicine Outperformed Other AI Tools

A powerful clinical artificial intelligence tool developed by University at Buffalo biomedical informatics researchers has demonstrated remarkable accuracy on all three parts of the United States Medical Licensing Exam (Step exams), according to a paper published in JAMA Network Open.

Achieving higher scores on the USMLE than most physicians and all other AI tools so far, Semantic Clinical Artificial Intelligence (SCAI, pronounced "Sky") has the potential to become a critical partner for physicians, says lead author Peter L. Elkin, MD, chair of the Department of Biomedical Informatics in the Jacobs School of Medicine and Biomedical Sciences at UB and a physician with UBMD Internal Medicine.

Elkin says SCAI is the most accurate clinical AI tool available to date, with the most advanced version scoring 95.2% on Step 3 of the USMLE, while a GPT4 Omni tool scored 90.5% on the same test.

"As physicians, we are used to using computers as tools," he explains, "but SCAI is different; it can add to your decision-making and thinking based on its own reasoning."

The tool can respond to medical questions posed by clinicians or the public at https://halsted.compbio.buffalo.edu/chat/.

The researchers tested the model against the USMLE, required for licensing physicians nationwide, which assesses the physician’s ability to apply knowledge, concepts and principles, and to demonstrate fundamental patient-centered skills. Any questions with a visual component were eliminated.

Elkin explains that most AI tools function by using statistics to find associations in online data that allow them to answer a question. "We call these tools generative artificial intelligence," he says. "Some have postulated that they are just plagiarizing what’s on the internet because the answers they give you are what others have written." However, these AI models are now becoming partners in care rather than simple tools for clinicians to utilize in their practice, he says.

"But SCAI answers more complex questions and performs more complex semantic reasoning," he says, "We have created knowledge sources that can reason more the way people learn to reason while doing their training in medical school."

The team started with a natural language processing software they had previously developed. They added vast amounts of authoritative clinical information gleaned from widely disparate sources ranging from recent medical literature and clinical guidelines to genomic data, drug information, discharge recommendations, patient safety data and more. Any data that might be biased, such as clinical notes, were not included.

SCAI contains 13 million medical facts, as well as all the possible interactions between those facts. The team used basic clinical facts known as semantic triples (subject-relation-object, such as “Penicillin treats pneumococcal pneumonia”) to create semantic networks. The tool can then represent these semantic networks so that it is possible to draw logical inferences from them.

"We have taught large language models how to use semantic reasoning," says Elkin.

Other techniques that contributed to SCAI include knowledge graphs that are designed to find new links in medical data as well as previously "hidden" patterns, as well as retrieval-augmented generation, which allows the large language model to access and incorporate information from external knowledge databases before responding to a prompt. This reduces "confabulation," the tendency for AI tools to always respond to a prompt even when it doesn’t have enough information to go on.

Elkin adds that using formal semantics to inform the large language model provides important context necessary for SCAI to understand and respond more accurately to a particular question.

"SCAI is different from other large language models because it can have a conversation with you and as a human-computer partnership can add to your decision-making and thinking based on its own reasoning," Elkin says.

He concludes: "By adding semantics to large language models, we are providing them with the ability to reason similarly to the way we do when practicing evidence-based medicine."

Because it can access such vast amounts of data, SCAI also has the potential to improve patient safety, improve access to care and "democratize specialty care," Elkin says, by making medical information on specialties and subspecialties accessible to primary care providers and even to patients.

While the power of SCAI is impressive, Elkin stresses its role will be to augment, not replace, physicians.

"Artificial intelligence isn’t going to replace doctors," he says, "but a doctor who uses AI may replace a doctor who does not."

Elkin PL, Mehta G, LeHouillier F, Resnick M, Mullin S, Tomlin C, Resendez S, Liu J, Nebeker JR, Brown SH.
Semantic Clinical Artificial Intelligence vs Native Large Language Model Performance on the USMLE.
JAMA Netw Open. 2025 Apr 1;8(4):e256359. doi: 10.1001/jamanetworkopen.2025.6359

Most Popular Now

Giving Doctors an AI-Powered Head Start …

Detection of melanoma and a range of other skin diseases will be faster and more accurate with a new artificial intelligence (AI) powered tool that analyses multiple imaging types simultaneously...

AI Agents for Oncology

Clinical decision-making in oncology is challenging and requires the analysis of various data types - from medical imaging and genetic information to patient records and treatment guidelines. To effectively support...

AI Medical Receptionist Modernizing Doct…

A virtual medical receptionist named "Cassie," developed through research at Texas A&M University, is transforming the way patients interact with health care providers. Cassie is a digital-human assistant created by Humanate...

Using Data and AI to Create Better Healt…

Academic medical centers could transform patient care by adopting principles from learning health systems principles, according to researchers from Weill Cornell Medicine and the University of California, San Diego. In...

AI Tool Set to Transform Characterisatio…

A multinational team of researchers, co-led by the Garvan Institute of Medical Research, has developed and tested a new AI tool to better characterise the diversity of individual cells within...

AI Detects Hidden Heart Disease Using Ex…

Mass General Brigham researchers have developed a new AI tool in collaboration with the United States Department of Veterans Affairs (VA) to probe through previously collected CT scans and identify...

Human-AI Collectives Make the Most Accur…

Diagnostic errors are among the most serious problems in everyday medical practice. AI systems - especially large language models (LLMs) like ChatGPT-4, Gemini, or Claude 3 - offer new ways...

Northern Ireland Completes Nationwide Ro…

Go-lives at Western and Southern health and social care trusts mean every pathology service is using the same laboratory information management system; improving efficiency and quality. An ambitious technology project to...

Highland Marketing Announced as Official…

Highland Marketing has been named, for the second year running, the official communications partner for HETT Show 2025, the UK's leading digital health conference and exhibition. Taking place 7-8 October...

MHP-Net: A Revolutionary AI Model for Ac…

Liver cancer is the sixth most common cancer globally and a leading cause of cancer-related deaths. Accurate segmentation of liver tumors is a crucial step for the management of the...

Groundbreaking TACIT Algorithm Offers Ne…

Researchers at VCU Massey Comprehensive Cancer Center have developed a novel algorithm that could provide a revolutionary tool for determining the best options for patients - both in the treatment...

The Many Ways that AI Enters Rheumatolog…

High-resolution computed tomography (HRCT) is the standard to diagnose and assess progression in interstitial lung disease (ILD), a key feature in systemic sclerosis (SSc). But AI-assisted interpretation has the potential...