Proposed Framework for Integrating Chatbots into Health Care

While the technology for developing artificial intelligence-powered chatbots has existed for some time, a new viewpoint piece in JAMA lays out the clinical, ethical, and legal aspects that must be considered before applying them in healthcare. And while the emergence of COVID-19 and the social distancing that accompanies it has prompted more health systems to explore and apply automated chatbots, the authors still urge caution and thoughtfulness before proceeding.

"We need to recognize that this is relatively new technology and even for the older systems that were in place, the data are limited," said the viewpoint's lead author, John D. McGreevey III, MD, an associate professor of Medicine in the Perelman School of Medicine at the University of Pennsylvania. "Any efforts also need to realize that much of the data we have comes from research, not widespread clinical implementation. Knowing that, evaluation of these systems must be robust when they enter the clinical space, and those operating them should be nimble enough to adapt quickly to feedback."

McGreevey, joined by C. William Hanson III, MD, chief medical information officer at Penn Medicine, and Ross Koppel, PhD, FACMI, a senior fellow at the Leonard Davis Institute of Healthcare Economics at Penn and professor of Medical Informatics, wrote "Clinical, Legal, and Ethical Aspects of AI-Assisted Conversational Agents." In it, the authors lay out 12 different focus areas that should be considered when planning to implement a chatbot, or, more formally, "conversational agent," in clinical care.

Chatbots are a tool used to communicate with patients via text message or voice. Many chatbots are powered by artificial intelligence (AI). This paper specifically discusses chatbots that use natural language processing, an AI process that seeks to "understand" language used in conversations and draws threads and connections from them to provide meaningful and useful answers.

With health care, those messages, and people's reactions to them, are extremely important and carry tangible consequences.

"We are increasingly in direct communication with our patients through electronic medical records, giving them direct access to their test results, diagnoses and doctors' notes," Hanson said. "Chatbots have the ability to enhance the value of those communications on the one hand, or cause confusion or even harm, on the other."

For instance, how a chatbot handles someone telling it something as serious as "I want to hurt myself" has many different implications.

In the self-harm example, there are several areas of focus laid out by the authors that apply. This touches first and foremost on the "Patient Safety" category: Who monitors the chatbot and how often do they do it? It also touches on "Trust and Transparency": Would this patient actually take a response from a known chatbot seriously? It also, unfortunately, raises questions in the paper's "Legal & Licensing" category: Who is accountable if the chatbot fails in its task. Moreover, a question under the "Scope" category may apply here, too: Is this a task best suited for a chatbot, or is it something that should still be totally human-operated?

Within their viewpoint, the team believes they have laid out key considerations that can inform a framework for decision-making when it comes to implementing chatbots in health care. Their considerations should apply even when rapid implementation is required to respond to events like the spread of COVID-19.

"To what extent should chatbots be extending the capabilities of clinicians, which we'd call augmented intelligence, or replacing them through totally artificial intelligence?" Koppel said. "Likewise, we need to determine the limits of chatbot authority to perform in different clinical scenarios, such as when a patient indicates that they have a cough, should the chatbot only respond by letting a nurse know or digging in further: 'Can you tell me more about your cough?'"

Chatbots have the opportunity to significantly improve health outcomes and lower health systems' operating costs, but evaluation and research will be key to that: both to ensure smooth operation and to keep the trust of both patients and health care workers.

"It's our belief that the work is not done when the conversational agent is deployed," McGreevey said. "These are going to be increasingly impactful technologies that deserve to be monitored not just before they are launched, but continuously throughout the life cycle of their work with patients."

Na-Na Zhang, Xiao-Feng Li, Yong-Qiang Deng, Hui Zhao, Yi-Jiao Huang, GuanYang, Wei-Jin Huang, Peng Gao, Chao Zhou, Rong-Rong Zhang, Yan Guo, Shi-Hui Sun, Hang Fan, Shu-Long Zu, Qi Chen, Qi He, Tian-Shu Cao, Xing-Yao Huang,Hong-Ying Qiu, Jian-Hui Nie, Yuhang Jiang, Hua-Yuan Yan, Qing Ye, Xia Zhong,Xia-Lin Xue, Zhen-Yu Zha, Dongsheng Zhou, Xiao Yang, You-Chun Wang, Bo Ying,Cheng-Feng Qin.
A thermostable mRNA vaccine against COVID-19.
Cell, 2020. doi: 10.1016/j.cell.2020.07.024

Most Popular Now

Philips Foundation 2024 Annual Report: E…

Marking its tenth anniversary, Philips Foundation released its 2024 Annual Report, highlighting a year in which the Philips Foundation helped provide access to quality healthcare for 46.5 million people around...

New AI Transforms Radiology with Speed, …

A first-of-its-kind generative AI system, developed in-house at Northwestern Medicine, is revolutionizing radiology - boosting productivity, identifying life-threatening conditions in milliseconds and offering a breakthrough solution to the global radiologist...

Scientists Argue for More FDA Oversight …

An agile, transparent, and ethics-driven oversight system is needed for the U.S. Food and Drug Administration (FDA) to balance innovation with patient safety when it comes to artificial intelligence-driven medical...

New Research Finds Specific Learning Str…

If data used to train artificial intelligence models for medical applications, such as hospitals across the Greater Toronto Area, differs from the real-world data, it could lead to patient harm...

Giving Doctors an AI-Powered Head Start …

Detection of melanoma and a range of other skin diseases will be faster and more accurate with a new artificial intelligence (AI) powered tool that analyses multiple imaging types simultaneously...

AI Agents for Oncology

Clinical decision-making in oncology is challenging and requires the analysis of various data types - from medical imaging and genetic information to patient records and treatment guidelines. To effectively support...

Patients say "Yes..ish" to the…

As artificial intelligence (AI) continues to be integrated in healthcare, a new multinational study involving Aarhus University sheds light on how dental patients really feel about its growing role in...

'AI Scientist' Suggests Combin…

An 'AI scientist', working in collaboration with human scientists, has found that combinations of cheap and safe drugs - used to treat conditions such as high cholesterol and alcohol dependence...

Brains vs. Bytes: Study Compares Diagnos…

A University of Maine study compared how well artificial intelligence (AI) models and human clinicians handled complex or sensitive medical cases. The study published in the Journal of Health Organization...

Start-ups in the Spotlight at MEDICA 202…

17 - 20 November 2025, Düsseldorf, Germany. MEDICA, the leading international trade fair and platform for healthcare innovations, will once again confirm its position as the world's number one hotspot for...