Could ChatGPT Help or Hurt Scientific Research Articles?

Since its introduction to the public in November 2022, ChatGPT, an artificial intelligence system, has substantially grown in use, creating written stories, graphics, art and more with just a short prompt from the user. But when it comes to scientific, peer-reviewed research, could the tool be useful?

“Right now, many journals do not want people to use ChatGPT to write their articles, but a lot of people are still trying to use it," said Melissa Kacena, PhD, vice chair of research and a professor of orthopaedic surgery at the Indiana University School of Medicine. "We wanted to study whether ChatGPT is able to write a scientific article and what are the different ways you could successfully use it."

The researchers took three different topics - fractures and the nervous system, Alzheimer's disease and bone health and COVID-19 and bone health - and prompted the subscription version of ChatGPT ($20/month) to create scientific articles about them. The researchers took 3 different approaches for the original draft of the articles - all human, all ChatGPT or a combination. The study is published in a compilation of 12 articles in a new, special edition of Current Osteoporosis Reports.

"The standard way of writing a review article is to do a literature search, write an outline, start writing, and then faculty members revise and edit the draft," Kacena said. "We collected data about how much time it takes for this human method and how much time it takes for ChatGPT to write and then for faculty to edit the different articles."

In the articles written only by ChatGPT, up to 70% of the references were wrong. But when using an AI-assisted approach with more human involvement, they saw more plagiarism, especially when giving the tool more references up front. Overall, the use of AI decreased time spent to write the article, but required more extensive fact checking.

Another concern is with the writing style used by ChatGPT. Even though the tool was prompted to use a higher level of scientific writing, the words and phrases were not necessarily written at the level someone would expect to see from a researcher.

"It was repetitive writing and even if it was structured the way you learn to write in school, it was scary to know there were maybe incorrect references or wrong information," said Lilian Plotkin, PhD, professor of anatomy, cell biology and physiology at the IU School of Medicine and coauthor on five of the papers.

Jill Fehrenbacher, PhD, associate professor of pharmacology and toxicology at the school and coauthor on nine of the papers, said she believes even though many scientific journals do not want authors to use ChatGPT, many people still will--especially non-native English speakers.

"People may still write everything themselves, but then put it into ChatGPT to fix their grammar or help with their writing, so I think we need to look at how do we shepherd people in using it appropriately and even helping them?" Fehrenbacher said. "We hope to provide a guide for the scientific community so that if people are going to use it, here are some tips and advice."

"I think it’s here to stay, but we need to understand how we can use it in an appropriate manner that won’t compromise someone’s reputation or spread misinformation," Kacena said.

Faculty and students from several departments and centers across the IU School of Medicine were involved, including orthopaedic surgery; anatomy, cell biology and physiology; pharmacology and toxicology; radiology and imaging sciences; anesthesia; the Stark Neuroscience Research Institute; the Indiana Center for Musculoskeletal Health; and the IU School of Dentistry. Authors are also affiliated with the Richard L. Roudebush Veterans Affairs Medical Center in Indianapolis, Eastern Virginia Medical School in Norfolk, Virginia, and Mount Holyoke College in South Hadley, Massachusetts.

Kacena MA, Plotkin LI, Fehrenbacher JC.
The Use of Artificial Intelligence in Writing Scientific Review Articles.
Curr Osteoporos Rep. 2024 Jan 16. doi: 10.1007/s11914-023-00852-0

Most Popular Now

AI in Personalized Cancer Medicine: New …

The application of AI in precision oncology has so far been largely confined to the development of new drugs and had only limited impact on the personalisation of therapies. New...

AI can Predict Brain Cancer Patients…

Artificial Intelligence (AI) can predict whether adult patients with brain cancer will survive more than eight months after receiving radiotherapy treatment. The use of the AI to successfully predict patient outcomes...

Max Planck Institute for Informatics and…

The Max Planck Institute for Informatics and Google deepen their strategic research partnership. With additional financial support from the U.S. IT company, the "Saarbrücken Research Center for Visual Computing, Interaction...

JMIR Medical Informatics Invites Submiss…

JMIR Publications has announced a new section titled, "AI Language Models in Health Care" in JMIR Medical Informatics. This leading peer-reviewed journal is indexed in PubMed and has a unique...

Paper Calls for Patient-First Regulation…

Ever wonder if the latest and greatest artificial intelligence (AI) tool you read about in the morning paper is going to save your life? A new study published in JAMA...

Could ChatGPT Help or Hurt Scientific Re…

Since its introduction to the public in November 2022, ChatGPT, an artificial intelligence system, has substantially grown in use, creating written stories, graphics, art and more with just a short...

Evaluating the Performance of AI-Based L…

A new study evaluates an artificial intelligence (AI)-based algorithm for autocontouring prior to radiotherapy in head and neck cancer. Manual contouring to pinpoint the area of treatment requires significant time...

Making AI a Partner in Neuroscientific D…

The past year has seen major advances in Large Language Models (LLMs) such as ChatGPT. The ability of these models to interpret and produce human text sources (and other sequence...

Chapman Scientists Code ChatGPT to Desig…

Generative artificial intelligence platforms, from ChatGPT to Midjourney, grabbed headlines in 2023. But GenAI can do more than create collaged images and help write emails - it can also design...

DMEA nova Award: Wanted - Visionary Solu…

9 - 11 April 2024, Berlin, Germany. The DMEA nova Award is being presented at DMEA 2024 for the first time. The award honours a digital health startup for an outstanding...

New Digital Therapy Reduces Anxiety and …

A therapist-guided digital cognitive behavioural therapy reduced distress in 89 per cent of participants living with long-term physical health conditions, a new King's College London study finds. Researchers at the Institute...

Europe's Digital Health Industry Me…

9 - 11 April 2024, Berlin, Germany. In just over two months, from 9 to 11 April 2024, DMEA, Europe's leading event for digitalisation of healthcare, will gather digital health experts...