Giving AI Penalties to Get Better Diagnoses

Anyone waiting for the results of a medical test knows the anxious question: 'Will my life change completely when I know?' And the relief if you test negative.

Nowadays, Artificial Intelligence (AI) is deployed more and more to predict life-threatening disease. But there remains a big challenge in getting the Machine Learning (ML) algorithms precise enough. Specifically, getting the algorithms to correctly diagnose if someone is sick.

Machine Learning (ML) is the branch of AI where algorithms learn from datasets and get smarter in the process.

"Let's say there is a dataset about a serious disease. The dataset has 90 people who do not have the disease. But 10 of the people do have the disease," says Dr Ibomoiye Domor Mienye. Mienye is a post-doctoral AI researcher at the University of Johannesburg (UJ).

"As an example, an ML algorithm says that the 90 do not have the disease. That is correct so far. But it fails to diagnose the 10 that do have the disease. The algorithm is still regarded as 90% accurate," he says.

This is because accuracy has been defined in this way. But for health outcomes, it may be urgent to diagnose the 10 people with the disease and get them into treatment. That may be more important than complete accuracy about the 90 who do not have the condition, he adds.

Penalties against AI

In a research study published in Informatics in Medicine Unlocked, Mienye and Prof Yanxia Sun show how ML algorithms can be improved significantly for medical purposes. They used logistic regression, decision tree, XGBoost, and random forest algorithms.

These are supervised binary classification algorithms. That means they only learn from the 'yes/no' datasets provided to them.

Dr Mienye and Prof Sun are both from the Department of Electrical and Engineering Science at UJ.

The researchers built cost sensitivity into each of the algorithms.

This means the algorithm gets a much bigger penalty for telling a sick person in the dataset that they are healthy, than the other way round. In medical terms, the algorithms get bigger penalties for false negatives than for false positives.

Disease datasets AI learns from

Dr Mienye and Prof Sun used public learning datasets for diabetes, breast cancer, cervical cancer (858 records) and chronic kidney disease (400 records).

The datasets come from large hospitals or healthcare programs. In these binary datasets, people are classified as either having a disease, or not having it at all.

The algorithms they used are binary also. These can say “yes the person has the disease” or “no they don’t have it.” They tested all the algorithms on each dataset, both without and with the cost-sensitivity.

Significantly improved precision and recall

The results make it clear that the penalties work as intended in these datasets.

For chronic kidney disease for example, the Random Forest algorithm had precision at 0.972 and recall at 0.946, out of a perfect 1.000.

After the cost-sensitivity was added, the algorithm improved significantly to precision at 0.990 and recall at a perfect 1.000.

For CKD, the three other algorithms’ recall improved from high scores to a perfect 1.000.

Precision at 1.000 means the algorithm did not predict one or more false positives across the entire dataset. Recall at 1.000 means the algorithm did not predict one or more false negatives across the entire dataset.

With the other datasets, the results were different for different algorithms.

For cervical cancer, the cost-sensitive Random Forest and XGBoost algorithms improved from high scores to perfect precision and recall. However, the Logistic Regression and Decision Tree algorithms improved to much higher scores but did not reach 1.000.

The precision problem

In general, algorithms have been more accurate at saying people do not have a disease, than identifying the ones who are sick, says Mienye. This is an ongoing challenge in healthcare AI.

The reason is the way the algorithms learn. The algorithms learn from datasets that come from large hospitals or state healthcare programs.

But most of the people in those datasets do not have the conditions they are being tested for, says Mienye.

"At a large hospital, a person comes in to get tested for chronic kidney disease (CKD). Their doctor sent them there because some of their symptoms are CKD symptoms. The doctor would like to rule out CKD. Turns out, the person does not have CKD.

"This happens with lots of people. The dataset ends up with more people who do not have CKD, than people who do. We call this an imbalanced dataset."

When an algorithm starts learning from the dataset, it learns far less about CKD than it should, and isn't accurate enough in diagnosing ill patients - unless the algorithm is adjusted for the imbalance.

AI on the other side of a boat ride

Mienye grew up in a village near the Atlantic Ocean, that is not accessible by road.

"You have to use a speedboat from the nearest town to get there. The boat ride takes two to three hours," he says.

The nearest clinic is in the bigger town, on the other side of the boat ride.

The deep rural setting of his home village inspired him to see how AI can help people with little or no access to healthcare.

An old lady from his village is a good example of how more advanced AI algorithms may assist in future, he says. A cost-sensitive multiclass ML algorithm could assess the measured data for her blood pressure, sodium levels, blood sugar and more.

If her data is recorded correctly on a computer, and the algorithm learns from a multiclass dataset, that future AI could tell clinic staff which stage of chronic kidney disease she is at.

This village scenario is in the future, however.

Meanwhile the study’s four algorithms with cost sensitivity, are far more precise at diagnosing disease in their numerical datasets.

And they learn quickly, using the ordinary computer that one could expect to find in a remote town.

Ibomoiye Domor Mienye, Yanxia Sun.
Performance analysis of cost-sensitive learning methods with application to imbalanced medical data.
Informatics in Medicine Unlocked, 2021. doi: 10.1016/j.imu.2021.100690

Most Popular Now

Artificial Intelligence in Healthcare Re…

This study presents an overview of the development, adoption and use of Artificial Intelligence (AI) technologies and applications in the healthcare sector across all Member States. The main aim of...

Giving AI Penalties to Get Better Diagno…

Anyone waiting for the results of a medical test knows the anxious question: 'Will my life change completely when I know?' And the relief if you test negative. Nowadays, Artificial Intelligence...

First-of-its-Kind Healthcare Communicati…

A web and mobile app-based start-up launched during the COVID-19 pandemic to help UK healthcare professionals communicate with patients has signed its first commercial agreement with a hospital trust...

New App Helps Parents Identify Treatable…

A ground-breaking new, mobile phone app, 'GrowthMonitor' places the accurate measurement of children's height in the hands of parents and carers. Preliminary data to be presented at the Society for...

DMEA Call for Papers: Digital Health in …

26 - 28 April 2022, Berlin, Germany. Health meets digitalisation. From 26 to 28 April 2022 DMEA - Connecting Digital Health, the leading platform for health IT, is opening its doors...

Health Meets IT: DMEA Newcomer Award Inv…

26 - 28 April 2022, Berlin, Germany. Digitalisation of our healthcare sector must be stepped up - the experience of recent months has shown that. It is why new ideas on...

Bittium Exhibits its High-Tech Medical T…

Bittium exhibits its innovative products and solutions for cardiology and neurophysiology as well as R&D services for the development of medical and healthcare technology at the MEDICA 2021 event. It...

Development of AI Technology for Produci…

Transcranial focused ultrasound can be used to treat degenerative movement disorders, intractable pain, and mental disorders by delivering ultrasound energy to a specific area of the brain without opening the...

Siemens Healthineers and UCSF Create Fir…

Siemens Healthineers and UC San Francisco have formed a research and innovation-driven collaboration to make radiological imaging greener, while improving access to and quality of radiological imaging in Northern California...

Tulane University Study Uses AI to Detec…

A Tulane University researcher found that artificial intelligence (AI) can accurately detect and diagnose colorectal cancer from tissue scans as well or better than pathologists, according to a new study...

Dedalus Acquires Swiftqueue to Support P…

Dedalus Group ("Dedalus"), a leading international healthcare software solutions provider, has announced to have completed the acquisition of 100% of Swiftqueue Technologies Ltd a fast-growing cloud-native appointment and scheduling solution...

FDA Authorizes Marketing of Virtual Real…

The U.S. Food and Drug Administration today authorized marketing of EaseVRx, a prescription-use immersive virtual reality (VR) system that uses cognitive behavioral therapy and other behavioral methods to help with...