Creating Exam Questions with ChatGPT

For the study, the UKB (Universit├Ątsklinikum Bonn) researchers created two sets of 25 multiple-choice questions (MCQs), each with five possible answers, one of which was correct. The first set of questions was written by an experienced medical lecturer, the second set was created by ChatGPT. 161 students answered all questions in random order. For each question, students also indicated whether they thought it was created by a human or by ChatGPT.

Matthias Laupichler, one of the study authors and research associate at the Institute for Medical Didactics at the UKB, explains: "We were surprised that the difficulty of human-generated and ChatGPT-generated questions was virtually identical. Even more surprising for us, however, was that the students were unable to correctly identify the origin of the question in almost half of the cases. Although the results obviously need to be replicated in further studies, the automated generation of exam questions using ChatGPT and co. appears to be a promising tool for medical studies."

His colleague and co-author of the study Johanna Rother adds: "Lecturers can use ChatGPT to generate ideas for exam questions, which are then checked and, if necessary, revised by the lecturers. In our opinion, however, students in particular benefit from the automated generation of medical practice questions, as it has long been known that self-testing one's own knowledge is very beneficial for learning."

Tobias Raupach, Director of the Institute of Medical Didactics, continues: "We knew from previous studies that language models such as ChatGPT can answer the questions in medical state examinations. We have now been able to show for the first time that the software can also be used to write new questions that hardly differ from those of experienced teachers."

Tizian Kaiser, who is studying human medicine in his seventh semester, comments: "When working on the mock exam, I was quite surprised at how difficult it was for me to tell the questions apart. My approach was to differentiate between the questions based on their length, the complexity of their sentence structure and the difficulty of their content. But to be honest, in some situations I simply had to guess and the evaluation showed that I was barely able to differentiate between them. This leads me to the conviction that a meaningful knowledge query, as in this exam, is also possible exclusively through questions posed by the AI."

He is convinced that ChatGPT has great potential for student learning. It allows students to repeat what they have learned in different ways and in different ways again and again. "There is the option of being quizzed by the AI on predefined topics, having mock exams designed or simulating oral exams in writing. The repetition of the material is thus tailored to the exam concept and the training possibilities are endless," says the study participant, while also qualifying: "However, I would only use Chat-GPT for this purpose and not beforehand in the learning process, in which the study topics have to be worked through and summarized. Because while Chat-GPT is excellent for repetition, I fear that errors can occur when preparing learning content. I wouldn't notice these errors without a prior overview of the topic."

It is known from other studies that regular testing - even and especially without grading - helps students to remember learning content more sustainably. Such tests can now be created with little effort. However, the current study should first be transferred to other contexts (i. e. other subjects, semesters and countries) and it should be investigated whether ChatGPT can also write questions other than the multiple choice questions commonly used in medicine.

Laupichler MC, Rother JF, Grunwald Kadow IC, Ahmadi S, Raupach T.
Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions.
Acad Med. 2023 Dec 28. doi: 10.1097/ACM.0000000000005626

Most Popular Now

AI in Personalized Cancer Medicine: New …

The application of AI in precision oncology has so far been largely confined to the development of new drugs and had only limited impact on the personalisation of therapies. New...

AI can Predict Brain Cancer Patients…

Artificial Intelligence (AI) can predict whether adult patients with brain cancer will survive more than eight months after receiving radiotherapy treatment. The use of the AI to successfully predict patient outcomes...

Max Planck Institute for Informatics and…

The Max Planck Institute for Informatics and Google deepen their strategic research partnership. With additional financial support from the U.S. IT company, the "Saarbr├╝cken Research Center for Visual Computing, Interaction...

JMIR Medical Informatics Invites Submiss…

JMIR Publications has announced a new section titled, "AI Language Models in Health Care" in JMIR Medical Informatics. This leading peer-reviewed journal is indexed in PubMed and has a unique...

Paper Calls for Patient-First Regulation…

Ever wonder if the latest and greatest artificial intelligence (AI) tool you read about in the morning paper is going to save your life? A new study published in JAMA...

Could ChatGPT Help or Hurt Scientific Re…

Since its introduction to the public in November 2022, ChatGPT, an artificial intelligence system, has substantially grown in use, creating written stories, graphics, art and more with just a short...

Evaluating the Performance of AI-Based L…

A new study evaluates an artificial intelligence (AI)-based algorithm for autocontouring prior to radiotherapy in head and neck cancer. Manual contouring to pinpoint the area of treatment requires significant time...

Making AI a Partner in Neuroscientific D…

The past year has seen major advances in Large Language Models (LLMs) such as ChatGPT. The ability of these models to interpret and produce human text sources (and other sequence...

Chapman Scientists Code ChatGPT to Desig…

Generative artificial intelligence platforms, from ChatGPT to Midjourney, grabbed headlines in 2023. But GenAI can do more than create collaged images and help write emails - it can also design...

DMEA nova Award: Wanted - Visionary Solu…

9 - 11 April 2024, Berlin, Germany. The DMEA nova Award is being presented at DMEA 2024 for the first time. The award honours a digital health startup for an outstanding...

New Digital Therapy Reduces Anxiety and …

A therapist-guided digital cognitive behavioural therapy reduced distress in 89 per cent of participants living with long-term physical health conditions, a new King's College London study finds. Researchers at the Institute...

Europe's Digital Health Industry Me…

9 - 11 April 2024, Berlin, Germany. In just over two months, from 9 to 11 April 2024, DMEA, Europe's leading event for digitalisation of healthcare, will gather digital health experts...