Creating Exam Questions with ChatGPT

For the study, the UKB (Universitätsklinikum Bonn) researchers created two sets of 25 multiple-choice questions (MCQs), each with five possible answers, one of which was correct. The first set of questions was written by an experienced medical lecturer, the second set was created by ChatGPT. 161 students answered all questions in random order. For each question, students also indicated whether they thought it was created by a human or by ChatGPT.

Matthias Laupichler, one of the study authors and research associate at the Institute for Medical Didactics at the UKB, explains: "We were surprised that the difficulty of human-generated and ChatGPT-generated questions was virtually identical. Even more surprising for us, however, was that the students were unable to correctly identify the origin of the question in almost half of the cases. Although the results obviously need to be replicated in further studies, the automated generation of exam questions using ChatGPT and co. appears to be a promising tool for medical studies."

His colleague and co-author of the study Johanna Rother adds: "Lecturers can use ChatGPT to generate ideas for exam questions, which are then checked and, if necessary, revised by the lecturers. In our opinion, however, students in particular benefit from the automated generation of medical practice questions, as it has long been known that self-testing one's own knowledge is very beneficial for learning."

Tobias Raupach, Director of the Institute of Medical Didactics, continues: "We knew from previous studies that language models such as ChatGPT can answer the questions in medical state examinations. We have now been able to show for the first time that the software can also be used to write new questions that hardly differ from those of experienced teachers."

Tizian Kaiser, who is studying human medicine in his seventh semester, comments: "When working on the mock exam, I was quite surprised at how difficult it was for me to tell the questions apart. My approach was to differentiate between the questions based on their length, the complexity of their sentence structure and the difficulty of their content. But to be honest, in some situations I simply had to guess and the evaluation showed that I was barely able to differentiate between them. This leads me to the conviction that a meaningful knowledge query, as in this exam, is also possible exclusively through questions posed by the AI."

He is convinced that ChatGPT has great potential for student learning. It allows students to repeat what they have learned in different ways and in different ways again and again. "There is the option of being quizzed by the AI on predefined topics, having mock exams designed or simulating oral exams in writing. The repetition of the material is thus tailored to the exam concept and the training possibilities are endless," says the study participant, while also qualifying: "However, I would only use Chat-GPT for this purpose and not beforehand in the learning process, in which the study topics have to be worked through and summarized. Because while Chat-GPT is excellent for repetition, I fear that errors can occur when preparing learning content. I wouldn't notice these errors without a prior overview of the topic."

It is known from other studies that regular testing - even and especially without grading - helps students to remember learning content more sustainably. Such tests can now be created with little effort. However, the current study should first be transferred to other contexts (i. e. other subjects, semesters and countries) and it should be investigated whether ChatGPT can also write questions other than the multiple choice questions commonly used in medicine.

Laupichler MC, Rother JF, Grunwald Kadow IC, Ahmadi S, Raupach T.
Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions.
Acad Med. 2023 Dec 28. doi: 10.1097/ACM.0000000000005626

Most Popular Now

AI-Powered CRISPR could Lead to Faster G…

Stanford Medicine researchers have developed an artificial intelligence (AI) tool to help scientists better plan gene-editing experiments. The technology, CRISPR-GPT, acts as a gene-editing “copilot” supported by AI to help...

Groundbreaking AI Aims to Speed Lifesavi…

To solve a problem, we have to see it clearly. Whether it’s an infection by a novel virus or memory-stealing plaques forming in the brains of Alzheimer’s patients, visualizing disease processes...

AI Spots Hidden Signs of Depression in S…

Depression is one of the most common mental health challenges, but its early signs are often overlooked. It is often linked to reduced facial expressivity. However, whether mild depression or...

ChatGPT 4o Therapeutic Chatbot 'Ama…

One of the first randomized controlled trials assessing the effectiveness of a large language model (LLM) chatbot 'Amanda' for relationship support shows that a single session of chatbot therapy...

AI Tools Help Predict Severe Asthma Risk…

Mayo Clinic researchers have developed artificial intelligence (AI) tools that help identify which children with asthma face the highest risk of serious asthma exacerbation and acute respiratory infections. The study...

AI Model Forecasts Disease Risk Decades …

Imagine a future where your medical history could help predict what health conditions you might face in the next two decades. Researchers have developed a generative AI model that uses...

AI Model Indicates Four out of Ten Breas…

A project at Lund University in Sweden has trained an AI model to identify breast cancer patients who could be spared from axillary surgery. The model analyses previously unutilised information...

AI Distinguishes Glioblastoma from Look-…

A Harvard Medical School–led research team has developed an AI tool that can reliably tell apart two look-alike cancers found in the brain but with different origins, behaviors, and treatments. The...

Overcoming the AI Applicability Crisis a…

Opinion Article by Harry Lykostratis, Chief Executive, Open Medical. The government’s 10 Year Health Plan makes a lot of the potential of AI-software to support clinical decision making, improve productivity, and...

Smart Device Uses AI and Bioelectronics …

As a wound heals, it goes through several stages: clotting to stop bleeding, immune system response, scabbing, and scarring. A wearable device called "a-Heal," designed by engineers at the University...

Dartford and Gravesham Implements Clinis…

Dartford and Gravesham NHS Trust has taken a significant step towards a more digital future by rolling out electronic test ordering using Clinisys ICE. The trust deployed the order communications...