Researchers Evaluate Accuracy of Online Health News Using Easily Accessible AI

It can be challenging to gauge the quality of online news - questioning if it is real or if it is fake. When it comes to health news and press releases about medical treatments and procedures the issue can be even more complex, especially if the story is not complete and still doesn’t necessarily fall into the category of fake news. To help identify the stories with inflated claims, inaccuracies and possible associated risks, researchers at the University of New Hampshire developed a new machine learning model, an application of artificial intelligence, that news services, like social media outlets, could easily use to better screen medical news stories for accuracy.

"The way most people think about fake news is something that's completely fabricated, but, especially in healthcare, it doesn't need to be fake. It could be that maybe they're not mentioning something," said Ermira Zifla, assistant professors of decision sciences at UNH's Peter T. Paul College of Business and Economics. "In the study, we're not making claims about the intent of the news organizations that put these out. But if things are left out, there should be a way to look at that."

Zifla and study co-author Burcu Eke Rubini, assistant professors of decision sciences, found in their research, published in Decision Support Systems, that since most people don’t have the medical expertise to understand the complexities of the news, the machine learning models they developed outperformed the evaluations of laypeople in assessing the quality of health stories. They used data from Health News Review that included news stories and press releases on new healthcare treatments published in various outlets from 2013 to 2018. The articles had already been evaluated by a panel of healthcare experts - medical doctors, healthcare journalists and clinical professors - using ten different evaluation criteria the experts had developed. The criteria included cost and benefits of the treatment or test, any possible harm, the quality of arguments, the novelty and availability of the procedure and the independence of the sources. The researchers then developed an algorithm based on the same expert criteria, and trained the machine models to classify each aspect of the news story, matching that criteria as "satisfactory" or "not satisfactory".

The model's performance was then compared against layperson evaluations obtained through a separate survey where participants rated the same articles as "satisfactory" or "not satisfactory" based on the same criteria. The survey revealed an "optimism bias," with most of the 254 participants rating articles as satisfactory, markedly different from the model's more critical assessments.

Researchers stress that they are by no means looking to replace expert opinion but are hoping to start a conversation about evaluating news based on multiple criteria and offering an easily accessible and low-cost alternative via open-source software to better evaluate health news.

Ermira Zifla, Burcu Eke Rubini.
Multi-criteria evaluation of health news stories.
Decision Support Systems, 2024. doi: 10.1016/j.dss.2024.114187

Most Popular Now

AI-Powered CRISPR could Lead to Faster G…

Stanford Medicine researchers have developed an artificial intelligence (AI) tool to help scientists better plan gene-editing experiments. The technology, CRISPR-GPT, acts as a gene-editing “copilot” supported by AI to help...

Groundbreaking AI Aims to Speed Lifesavi…

To solve a problem, we have to see it clearly. Whether it’s an infection by a novel virus or memory-stealing plaques forming in the brains of Alzheimer’s patients, visualizing disease processes...

AI Spots Hidden Signs of Depression in S…

Depression is one of the most common mental health challenges, but its early signs are often overlooked. It is often linked to reduced facial expressivity. However, whether mild depression or...

AI Tools Help Predict Severe Asthma Risk…

Mayo Clinic researchers have developed artificial intelligence (AI) tools that help identify which children with asthma face the highest risk of serious asthma exacerbation and acute respiratory infections. The study...

ChatGPT 4o Therapeutic Chatbot 'Ama…

One of the first randomized controlled trials assessing the effectiveness of a large language model (LLM) chatbot 'Amanda' for relationship support shows that a single session of chatbot therapy...

AI Model Forecasts Disease Risk Decades …

Imagine a future where your medical history could help predict what health conditions you might face in the next two decades. Researchers have developed a generative AI model that uses...

AI Model Indicates Four out of Ten Breas…

A project at Lund University in Sweden has trained an AI model to identify breast cancer patients who could be spared from axillary surgery. The model analyses previously unutilised information...

AI Distinguishes Glioblastoma from Look-…

A Harvard Medical School–led research team has developed an AI tool that can reliably tell apart two look-alike cancers found in the brain but with different origins, behaviors, and treatments. The...

Overcoming the AI Applicability Crisis a…

Opinion Article by Harry Lykostratis, Chief Executive, Open Medical. The government’s 10 Year Health Plan makes a lot of the potential of AI-software to support clinical decision making, improve productivity, and...

Smart Device Uses AI and Bioelectronics …

As a wound heals, it goes through several stages: clotting to stop bleeding, immune system response, scabbing, and scarring. A wearable device called "a-Heal," designed by engineers at the University...

Dartford and Gravesham Implements Clinis…

Dartford and Gravesham NHS Trust has taken a significant step towards a more digital future by rolling out electronic test ordering using Clinisys ICE. The trust deployed the order communications...