An AI Tool Grounded in Evidence-Based Medicine Outperformed Other AI Tools

A powerful clinical artificial intelligence tool developed by University at Buffalo biomedical informatics researchers has demonstrated remarkable accuracy on all three parts of the United States Medical Licensing Exam (Step exams), according to a paper published in JAMA Network Open.

Achieving higher scores on the USMLE than most physicians and all other AI tools so far, Semantic Clinical Artificial Intelligence (SCAI, pronounced "Sky") has the potential to become a critical partner for physicians, says lead author Peter L. Elkin, MD, chair of the Department of Biomedical Informatics in the Jacobs School of Medicine and Biomedical Sciences at UB and a physician with UBMD Internal Medicine.

Elkin says SCAI is the most accurate clinical AI tool available to date, with the most advanced version scoring 95.2% on Step 3 of the USMLE, while a GPT4 Omni tool scored 90.5% on the same test.

"As physicians, we are used to using computers as tools," he explains, "but SCAI is different; it can add to your decision-making and thinking based on its own reasoning."

The tool can respond to medical questions posed by clinicians or the public at https://halsted.compbio.buffalo.edu/chat/.

The researchers tested the model against the USMLE, required for licensing physicians nationwide, which assesses the physician’s ability to apply knowledge, concepts and principles, and to demonstrate fundamental patient-centered skills. Any questions with a visual component were eliminated.

Elkin explains that most AI tools function by using statistics to find associations in online data that allow them to answer a question. "We call these tools generative artificial intelligence," he says. "Some have postulated that they are just plagiarizing what’s on the internet because the answers they give you are what others have written." However, these AI models are now becoming partners in care rather than simple tools for clinicians to utilize in their practice, he says.

"But SCAI answers more complex questions and performs more complex semantic reasoning," he says, "We have created knowledge sources that can reason more the way people learn to reason while doing their training in medical school."

The team started with a natural language processing software they had previously developed. They added vast amounts of authoritative clinical information gleaned from widely disparate sources ranging from recent medical literature and clinical guidelines to genomic data, drug information, discharge recommendations, patient safety data and more. Any data that might be biased, such as clinical notes, were not included.

SCAI contains 13 million medical facts, as well as all the possible interactions between those facts. The team used basic clinical facts known as semantic triples (subject-relation-object, such as “Penicillin treats pneumococcal pneumonia”) to create semantic networks. The tool can then represent these semantic networks so that it is possible to draw logical inferences from them.

"We have taught large language models how to use semantic reasoning," says Elkin.

Other techniques that contributed to SCAI include knowledge graphs that are designed to find new links in medical data as well as previously "hidden" patterns, as well as retrieval-augmented generation, which allows the large language model to access and incorporate information from external knowledge databases before responding to a prompt. This reduces "confabulation," the tendency for AI tools to always respond to a prompt even when it doesn’t have enough information to go on.

Elkin adds that using formal semantics to inform the large language model provides important context necessary for SCAI to understand and respond more accurately to a particular question.

"SCAI is different from other large language models because it can have a conversation with you and as a human-computer partnership can add to your decision-making and thinking based on its own reasoning," Elkin says.

He concludes: "By adding semantics to large language models, we are providing them with the ability to reason similarly to the way we do when practicing evidence-based medicine."

Because it can access such vast amounts of data, SCAI also has the potential to improve patient safety, improve access to care and "democratize specialty care," Elkin says, by making medical information on specialties and subspecialties accessible to primary care providers and even to patients.

While the power of SCAI is impressive, Elkin stresses its role will be to augment, not replace, physicians.

"Artificial intelligence isn’t going to replace doctors," he says, "but a doctor who uses AI may replace a doctor who does not."

Elkin PL, Mehta G, LeHouillier F, Resnick M, Mullin S, Tomlin C, Resendez S, Liu J, Nebeker JR, Brown SH.
Semantic Clinical Artificial Intelligence vs Native Large Language Model Performance on the USMLE.
JAMA Netw Open. 2025 Apr 1;8(4):e256359. doi: 10.1001/jamanetworkopen.2025.6359

Most Popular Now

AI also Assesses Dutch Mammograms Better…

AI is detecting tumors more often and earlier in the Dutch breast cancer screening program. Those tumors can then be treated at an earlier stage. This has been demonstrated by...

Unlocking the 10 Year Health Plan

The government's plan for the NHS is a huge document. Jane Stephenson, chief executive of SPARK TSL, argues the key to unlocking its digital ambitions is to consider what it...

AI can Find Cancer Pathologists Miss

Men assessed as healthy after a pathologist analyses their tissue sample may still have an early form of prostate cancer. Using AI, researchers at Uppsala University have been able to...

Alcidion Grows Top Talent in the UK, wit…

Alcidion has today announced the addition of three new appointments to their UK-based team, with one internal promotion and two external recruits. Dr Paul Deffley has been announced as the...

How AI could Speed the Development of RN…

Using artificial intelligence (AI), MIT researchers have come up with a new way to design nanoparticles that can more efficiently deliver RNA vaccines and other types of RNA therapies. After training...

AI, Full Automation could Expand Artific…

Automated insulin delivery (AID) systems such as the UVA Health-developed artificial pancreas could help more type 1 diabetes patients if the devices become fully automated, according to a new review...

MIT Researchers Use Generative AI to Des…

With help from artificial intelligence, MIT researchers have designed novel antibiotics that can combat two hard-to-treat infections: drug-resistant Neisseria gonorrhoeae and multi-drug-resistant Staphylococcus aureus (MRSA). Using generative AI algorithms, the research...

Penn Developed AI Tools and Datasets Hel…

Doctors treating kidney disease have long depended on trial-and-error to find the best therapies for individual patients. Now, new artificial intelligence (AI) tools developed by researchers in the Perelman School...

AI Hybrid Strategy Improves Mammogram In…

A hybrid reading strategy for screening mammography, developed by Dutch researchers and deployed retrospectively to more than 40,000 exams, reduced radiologist workload by 38% without changing recall or cancer detection...

New Training Year Starts at Siemens Heal…

In September, 197 school graduates will start their vocational training or dual studies in Germany at Siemens Healthineers. 117 apprentices and 80 dual students will begin their careers at Siemens...

Routine AI Assistance may Lead to Loss o…

The introduction of artificial intelligence (AI) to assist colonoscopies is linked to a reduction in the ability of endoscopists (health professionals who perform colonoscopies) to detect precancerous growths (adenomas) in...

New AI Tool Addresses Accuracy and Fairn…

A team of researchers at the Icahn School of Medicine at Mount Sinai has developed a new method to identify and reduce biases in datasets used to train machine-learning algorithms...