ChatGPT Helpful for Breast Cancer Screening Advice with Certain Caveats

As more consumers turn to the newly available ChatGPT for health advice, researchers are eager to see whether the information provided by the artificial intelligence chatbot is reliable and accurate. A new study conducted by researchers at the University of Maryland School of Medicine (UMSOM) indicates that the answers generated provide correct information the vast majority of the time; sometimes, though, the information is inaccurate or even fictitious.

Findings were published today in the journal Radiology.

In February 2023, UMSOM researchers created a set of 25 questions related to advice on getting screened for breast cancer. They submitted each question to ChatGPT three times to see what responses were generated. (The chatbot is known for varying its response each time a question is posed.) Three radiologists fellowship-trained in mammography evaluated the responses; they found that the responses were appropriate for 22 out of the 25 questions. The chatbot did, however, provide one answer based on outdated information. Two other questions had inconsistent responses that varied significantly each time the same question was posed.

"We found ChatGPT answered questions correctly about 88 percent of the time, which is pretty amazing," said study corresponding author Paul Yi , MD, Assistant Professor of Diagnostic Radiology and Nuclear Medicine at UMSOM and Director of the UM Medical Intelligent Imaging Center (UM2ii). "It also has the added benefit of summarizing information into an easily digestible form for consumers to easily understand." ChatGPT correctly answered questions about the symptoms of breast cancer, who is at risk, and questions on the cost, age, and frequency recommendations concerning mammograms.

The downside is that it is not as comprehensive in its responses as what a person would normally find on a Google search. "ChatGPT provided only one set of recommendations on breast cancer screening, issued from the American Cancer Society, but did not mention differing recommendations put out by the Centers for Disease Control and Prevention (CDC) or the US Preventative Services Task Force (USPSTF)," said study lead author Hana Haver, MD, a radiology resident at University of Maryland Medical Center.

In one response deemed by the researchers to be inappropriate, ChatGPT provided an outdated response to planning a mammogram around COVID-19 vaccination. The advice to delay a mammogram for four to six weeks after getting a COVID-19 shot was changed in February 2022, and the CDC endorses the USPSTF guidelines, which don’t recommend waiting. Inconsistent responses were given to questions concerning an individual's personal risk of getting breast cancer and on where someone could get a mammogram.

"We've seen in our experience that ChatGPT sometimes makes up fake journal articles or health consortiums to support its claims," said Dr. Yi. "Consumers should be aware that these are new, unproven technologies, and should still rely on their doctor, rather than ChatGPT, for advice."

He and his colleagues are now analyzing how ChatGPT fares for lung cancer screening recommendations and identifying ways to improve the recommendations made by ChatGPT to be more accurate and complete - as well as understandable to those without a high level of education.

"With the rapid evolution of ChatGPT and other large language models, we have a responsibility as a medical community to evaluate these technologies and protect our patients from potential harm that may come from incorrect screening recommendations or outdated preventive health strategies," said Mark T. Gladwin, MD, Dean, University of Maryland School of Medicine, Vice President for Medical Affairs, University of Maryland, Baltimore, and the John Z. and Akiko K. Bowers Distinguished Professor.

Haver HL, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J, Yi PH.
Appropriateness of Breast Cancer Prevention and Screening Recommendations Provided by ChatGPT.
Radiology. 2023 Apr 4:230424. doi: 10.1148/radiol.230424

Most Popular Now

AI Predictions for Colorectal Cancer: On…

Colorectal cancer (CRC) ranks second in leading causes of cancer-related deaths globally, according to the WHO. For the first time, researchers from Helmholtz Munich and the University of Technology Dresden...

Combining AI Models Improves Breast Canc…

Combining artificial intelligence (AI) systems for short- and long-term breast cancer risk results in an improved cancer risk assessment, according to a study published in Radiology, a journal of the...

ChatGPT Shows 'Impressive' Acc…

A new study led by investigators from Mass General Brigham has found that ChatGPT was about 72 percent accurate in overall clinical decision making, from coming up with possible diagnoses...

Healthcare Chatbot: Expand Support with …

The Danish eHealth platform, sundhed.dk, has faced a substantial surge in requests from Danish citizens and has swiftly expanded its support and effectively adapt to the ongoing changes in queries due...

WiFi SPARK's Healthcare Business Re…

Leading WiFi provider WiFi SPARK is rebranding its healthcare arm as SPARK Technology Services Limited. The new identity marks the completion of the integration of the former Hospedia bedside unit...

ChatGPT is Debunking Myths on Social Med…

ChatGPT could help to increase vaccine uptake by debunking myths around jab safety, say the authors of a study published in the peer-reviewed journal Human Vaccines and Immunotherapeutics. The researchers asked...

Online AI-Based Test for Parkinson'…

An artificial intelligence (AI) tool developed by researchers at the University of Rochester can help people with Parkinson's disease remotely assess the severity of their symptoms within minutes. A study...

AI Performs Comparably to Human Readers …

Using a standardized assessment, researchers in the UK compared the performance of a commercially available artificial intelligence (AI) algorithm with human readers of screening mammograms. Results of their findings were...

Siemens Healthineers Expands Production …

Siemens Healthineers is expanding its site in Rudolstadt, Germany. By mid 2024, a new manufacturing building will be built on the site. The new manufacturing plant will produce electron accelerators...

More Cases of Breast Cancer Detected wit…

One radiologist supported by AI detected more cases of breast cancer in screening mammography than two radiologists working together, reports the ScreenTrustCAD study from Karolinska Institutet in The Lancet Digital...

MEDICA 2023 + COMPAMED 2023: "Where…

13 - 16 November 2023, Düsseldorf, Germany. The medical technology market is in worldwide motion and the signs ahead of MEDICA 2023 and COMPAMED 2023 in Düsseldorf as the internationally leading...

Smartphone Technology Expected to Advanc…

Since the 1980s, we have known that neurological soft signs (NSS) can distinguish people with schizophrenia from psychiatrically healthy individuals. NSS are subtle neurological impairments that principally manifest as decreased...