ChatGPT Shows Promise in Answering Patients' Questions to Urologists

The groundbreaking ChatGPT chatbot shows potential as a time-saving tool for responding to patient questions sent to the urologist's office, suggests a study in the September issue of Urology Practice®, an Official Journal of the American Urological Association (AUA). The journal is published in the Lippincott portfolio by Wolters Kluwer.

The artificial intelligence (AI) tool generated "acceptable" responses to nearly one-half of a sample of real-life patient questions, according to the new research by Michael Scott, MD, a urologist at Stanford University School of Medicine. "Generative AI technologies may play a valuable role in providing prompt, accurate responses to routine patient questions - potentially alleviating patients' concerns while freeing up clinic time and resources to address other complex tasks," Dr. Scott comments.

Can ChatGPT accurately answer questions from urology patients?

ChatGPT is an innovative large language model (LLM) that has sparked interest across a wide range of settings, including health and medicine. In some recent studies, ChatGPT has performed well in responding to various types of medical questions, although its performance in urology is less well-established.

Modern electronic health record (EHR) systems enable patients to send medical questions directly to their doctors. "This shift has been associated with an increased time burden of EHR use for physicians with a large portion of this attributed to patient in-basket messages," the researchers write. One study estimates that each message in a physician's inbox adds more than two minutes spent on the EHR.

Dr. Scott and colleagues collected 100 electronic patient messages requesting medical advice from a urologist at a men's health clinic. The messages were categorized by type of content and difficulty, then entered into ChatGPT. Five experienced urologists graded each AI-generated response in terms of accuracy, completeness, helpfulness, and intelligibility. Raters also indicated whether they would send each response to a patient.

Findings support 'generative AI technology to improve clinical efficiency'

The ChatGPT-generated responses were judged to be accurate, with an average score of 4.0 on a five-point scale; and intelligible, average score 4.7. Ratings of completeness and helpfulness were lower, but with little or no potential for harm. Scores were comparable for different types of question content (symptoms, postoperative concerns, etc).

"Overall, 47% of responses were deemed acceptable to send to patients," the researchers write. Questions rated as "easy" had a higher rate of acceptable responses: 56%, compared to 34% for "difficult" questions.

"These results show promise for the utilization of generative AI technology to help improve clinical efficiency," Dr. Scott and coauthors write. The findings "suggest the feasibility of integrating this new technology into clinical care to improve efficiency while maintaining quality of patient communication."

The researchers note some potential drawbacks of ChatGPT-generated responses to patient questions: "ChatGPT's model is trained on information from the Internet in general, as opposed to validated medical sources," with a "risk of generating inaccurate or misleading responses." The authors also highlight the need for safeguards to ensure patient privacy.

"While our study provides an interesting starting point, more research will be needed to validate the use of LLMs to respond to patient questions, in urology as well as other specialties," Dr. Scott comments. "This will be a potentially valuable healthcare application, particularly with continued advances in AI technology."

Scott M, Muncey W, Seranio N, Belladelli F, Del Giudice F, Li S, Ha A, Glover F, Zhang CA, Eisenberg ML.
Assessing Artificial Intelligence-Generated Responses to Urology Patient In-Basket Messages.
Urol Pract. 2024 Sep;11(5):793-798. doi: 10.1097/UPJ.0000000000000637

Most Popular Now

Unlocking the 10 Year Health Plan

The government's plan for the NHS is a huge document. Jane Stephenson, chief executive of SPARK TSL, argues the key to unlocking its digital ambitions is to consider what it...

Alcidion Grows Top Talent in the UK, wit…

Alcidion has today announced the addition of three new appointments to their UK-based team, with one internal promotion and two external recruits. Dr Paul Deffley has been announced as the...

AI can Find Cancer Pathologists Miss

Men assessed as healthy after a pathologist analyses their tissue sample may still have an early form of prostate cancer. Using AI, researchers at Uppsala University have been able to...

New Training Year Starts at Siemens Heal…

In September, 197 school graduates will start their vocational training or dual studies in Germany at Siemens Healthineers. 117 apprentices and 80 dual students will begin their careers at Siemens...

AI, Full Automation could Expand Artific…

Automated insulin delivery (AID) systems such as the UVA Health-developed artificial pancreas could help more type 1 diabetes patients if the devices become fully automated, according to a new review...

How AI could Speed the Development of RN…

Using artificial intelligence (AI), MIT researchers have come up with a new way to design nanoparticles that can more efficiently deliver RNA vaccines and other types of RNA therapies. After training...

MIT Researchers Use Generative AI to Des…

With help from artificial intelligence, MIT researchers have designed novel antibiotics that can combat two hard-to-treat infections: drug-resistant Neisseria gonorrhoeae and multi-drug-resistant Staphylococcus aureus (MRSA). Using generative AI algorithms, the research...

AI Hybrid Strategy Improves Mammogram In…

A hybrid reading strategy for screening mammography, developed by Dutch researchers and deployed retrospectively to more than 40,000 exams, reduced radiologist workload by 38% without changing recall or cancer detection...

Penn Developed AI Tools and Datasets Hel…

Doctors treating kidney disease have long depended on trial-and-error to find the best therapies for individual patients. Now, new artificial intelligence (AI) tools developed by researchers in the Perelman School...

Are You Eligible for a Clinical Trial? C…

A new study in the academic journal Machine Learning: Health discovers that ChatGPT can accelerate patient screening for clinical trials, showing promise in reducing delays and improving trial success rates. Researchers...

Global Study Reveals How Patients View M…

How physicians feel about artificial intelligence (AI) in medicine has been studied many times. But what do patients think? A team led by researchers at the Technical University of Munich...

New AI Tool Addresses Accuracy and Fairn…

A team of researchers at the Icahn School of Medicine at Mount Sinai has developed a new method to identify and reduce biases in datasets used to train machine-learning algorithms...