Generative AI's Diagnostic Capabilities Comparable to Non-Dpecialist Doctors

The use of generative AI for diagnostics has attracted attention in the medical field and many research papers have been published on this topic. However, because the evaluation criteria were different for each study, a comprehensive analysis was needed to determine the extent AI could be used in actual medical settings and what advantages it featured in comparison to doctors.

A research group led by Dr. Hirotaka Takita and Associate Professor Daiju Ueda at Osaka Metropolitan University’s Graduate School of Medicine conducted a meta-analysis of generative AI's diagnostic capabilities using 83 research papers published between June 2018 and June 2024 that covered a wide range of medical specialties. Of the large language models (LLMs) that were analyzed, ChatGPT was the most commonly studied.

The comparative evaluation revealed that medical specialists had a 15.8% higher diagnostic accuracy than generative AI. The average diagnostic accuracy of generative AI was 52.1%, with the latest models of generative AI sometimes showing accuracy on par with non-specialist doctors.

"This research shows that generative AI’s diagnostic capabilities are comparable to non-specialist doctors. It could be used in medical education to support non-specialist doctors and assist in diagnostics in areas with limited medical resources." stated Dr. Takita. "Further research, such as evaluations in more complex clinical scenarios, performance evaluations using actual medical records, improving the transparency of AI decision-making, and verification in diverse patient groups, is needed to verify AI’s capabilities."

Takita H, Kabata D, Walston SL, Tatekawa H, Saito K, Tsujimoto Y, Miki Y, Ueda D.
A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians.
NPJ Digit Med. 2025 Mar 22;8(1):175. doi: 10.1038/s41746-025-01543-z

Most Popular Now

AI Tool Helps Predict Relapse of Pediatr…

Artificial intelligence (AI) shows tremendous promise for analyzing vast medical imaging datasets and identifying patterns that may be missed by human observers. AI-assisted interpretation of brain scans may help improve...

Detecting Lung Cancer 4 Months Earlier a…

GPs may soon be able to identify patients with an increased risk of lung cancer up to 4 months earlier than is currently the case. The GP should be able...

Infectious Disease Surveillance Platform…

The Biothreats Emergence, Analysis and Communications Network (BEACON) leverages advanced artificial intelligence (AI), large language models (LLMs) and a network of globally based experts to rapidly collect, analyze, and disseminate...

Children's Health Ireland to Transf…

Healthcare teams responsible for paediatric care in Ireland are to save significant time in accessing important diagnostic imaging and reports, with the help of a new agreement with medical imaging...

NHS, Councils, and Housing could Share N…

A new technology partnership formally announced, could help NHS, local government, and housing organisations collaborate to create an unprecedented understanding of the risks and needs of people in their care...

AI-Powered Analysis of Stent Healing

Each year, more than three million people worldwide are treated with stents to open blocked blood vessels caused by heart disease. However, monitoring the healing process after implantation remains a...

Right Patient, Right Dose, Right Time

While artificial intelligence (AI) has shown promising potential, much of its use has remained theoretical or retrospective. Turning its potential into real-world healthcare outcomes, researchers at the Yong Loo Lin...

An AI Tool Grounded in Evidence-Based Me…

A powerful clinical artificial intelligence tool developed by University at Buffalo biomedical informatics researchers has demonstrated remarkable accuracy on all three parts of the United States Medical Licensing Exam (Step...

AXREM and BHTA Name Highland as 'Fu…

Hosted by trade associations AXREM and the British Healthcare Trades Association (BHTA), 'The Future of MedTech - Innovating for Tomorrow', will allow delegates to engage with speakers from the government...