When it comes to Emergency Care, ChatGPT Overprescribes

If ChatGPT were cut loose in the Emergency Department, it might suggest unneeded x-rays and antibiotics for some patients and admit others who didn't require hospital treatment, a new study from UC San Francisco has found.

The researchers said that, while the model could be prompted in ways that make its responses more accurate, it's still no match for the clinical judgment of a human doctor.

"This is a valuable message to clinicians not to blindly trust these models," said postdoctoral scholar Chris Williams, MB BChir, lead author of the study, which appears Oct. 8 in Nature Communications. "ChatGPT can answer medical exam questions and help draft clinical notes, but it’s not currently designed for situations that call for multiple considerations, like the situations in an emergency department."

Recently, Williams showed that ChatGPT, a large language model (LLM) that can be used for researching clinical applications of AI, was slightly better than humans at determining which of two emergency patients was most acutely unwell, a straightforward choice between patient A and patient B.

With the current study, Williams challenged the AI model to perform a more complex task: providing the recommendations a physician makes after initially examining a patient in the ED. This includes deciding whether to admit the patient, get x-rays or other scans, or prescribe antibiotics.

For each of the three decisions, the team compiled a set of 1,000 ED visits to analyze from an archive of more than 251,000 visits. The sets had the same ratio of “yes” to “no” responses for decisions on admission, radiology and antibiotics that are seen across UCSF Health’s Emergency Department.

Using UCSF’s secure generative AI platform, which has broad privacy protections, the researchers entered doctors’ notes on each patient’s symptoms and examination findings into ChatGPT-3.5 and ChatGPT-4. Then, they tested the accuracy of each set with a series of increasingly detailed prompts.

Overall, the AI models tended to recommend services more often than was needed. ChatGPT-4 was 8% less accurate than resident physicians, and ChatGPT-3.5 was 24% less accurate.

Williams said the AI’s tendency to overprescribe could be because the models are trained on the internet, where legitimate medical advice sites aren’t designed to answer emergency medical questions but rather to send readers to a doctor who can.

"These models are almost fine-tuned to say, 'seek medical advice,' which is quite right from a general public safety perspective," he said. "But erring on the side of caution isn’t always appropriate in the ED setting, where unnecessary interventions could cause patients harm, strain resources and lead to higher costs for patients."

He said models like ChatGPT will need better frameworks for evaluating clinical information before they are ready for the ED. The people who design those frameworks will need to strike a balance between making sure the AI doesn't miss something serious, while keeping it from triggering unneeded exams and expenses.

This means researchers developing medical applications of AI, along with the wider clinical community and the public, need to consider where to draw those lines and how much to err on the side of caution.

"There's no perfect solution," he said, "But knowing that models like ChatGPT have these tendencies, we’re charged with thinking through how we want them to perform in clinical practice."

Williams CYK, Miao BY, Kornblith AE, Butte AJ.
Evaluating the use of large language models to provide clinical recommendations in the Emergency Department.
Nat Commun. 2024 Oct 8;15(1):8236. doi: 10.1038/s41467-024-52415-1

Most Popular Now

AI Catches One-Third of Interval Breast …

An AI algorithm for breast cancer screening has potential to enhance the performance of digital breast tomosynthesis (DBT), reducing interval cancers by up to one-third, according to a study published...

Great plan: Now We need to Get Real abou…

The government's big plan for the 10 Year Health Plan for the NHS laid out a big role for delivery. However, the Highland Marketing advisory board felt the missing implementation...

Researchers Create 'Virtual Scienti…

There may be a new artificial intelligence-driven tool to turbocharge scientific discovery: virtual labs. Modeled after a well-established Stanford School of Medicine research group, the virtual lab is complete with an...

From WebMD to AI Chatbots: How Innovatio…

A new research article published in the Journal of Participatory Medicine unveils how successive waves of digital technology innovation have empowered patients, fostering a more collaborative and responsive health care...

New AI Tool Accelerates mRNA-Based Treat…

A new artificial intelligence (AI) model can improve the process of drug and vaccine discovery by predicting how efficiently specific mRNA sequences will produce proteins, both generally and in various...

AI also Assesses Dutch Mammograms Better…

AI is detecting tumors more often and earlier in the Dutch breast cancer screening program. Those tumors can then be treated at an earlier stage. This has been demonstrated by...

RSNA AI Challenge Models can Independent…

Algorithms submitted for an AI Challenge hosted by the Radiological Society of North America (RSNA) have shown excellent performance for detecting breast cancers on mammography images, increasing screening sensitivity while...

AI could Help Emergency Rooms Predict Ad…

Artificial intelligence (AI) can help emergency department (ED) teams better anticipate which patients will need hospital admission, hours earlier than is currently possible, according to a multi-hospital study by the...

Head-to-Head Against AI, Pharmacy Studen…

Students pursuing a Doctor of Pharmacy degree routinely take - and pass - rigorous exams to prove competency in several areas. Can ChatGPT accurately answer the same questions? A new...

NHS Active 10 Walking Tracker Users are …

Users of the NHS Active 10 app, designed to encourage people to become more active, immediately increased their amount of brisk and non-brisk walking upon using the app, according to...

New AI Tool Illuminates "Dark Side…

Proteins sustain life as we know it, serving many important structural and functional roles throughout the body. But these large molecules have cast a long shadow over a smaller subclass...

Deep Learning-Based Model Enables Fast a…

Stroke is the second leading cause of death globally. Ischemic stroke, strongly linked to atherosclerotic plaques, requires accurate plaque and vessel wall segmentation and quantification for definitive diagnosis. However, conventional...