AI for Real-Rime, Patient-Focused Insight

A picture may be worth a thousand words, but still... they both have a lot of work to do to catch up to BiomedGPT.

Covered recently in the prestigious journal Nature Medicine, BiomedGPT is a new a new type of artificial intelligence (AI) designed to support a wide range of medical and scientific tasks. This new study, conducted in collaboration with multiple institutions, is described in the article as "the first open-source and lightweight vision–language foundation model, designed as a generalist capable of performing various biomedical tasks."

"This work combines two types of AI into a decision support tool for medical providers," explains Lichao Sun, an assistant professor of computer science and engineering at Lehigh University and a lead author of the study. "One side of the system is trained to understand biomedical images, and one is trained to understand and assess biomedical text. The combination of these allows the model to tackle a wide range of biomedical challenges, using insight gleaned from databases of biomedical imagery and from the analysis and synthesis of scientific and medical research reports."

The key innovation described in the August 7 Nature Medicine article, “A generalist vision–language foundation model for diverse biomedical tasks,” is that this AI model doesn’t need to be specialized for each task. Typically, AI systems are trained for specific jobs, like recognizing tumors in X-rays or summarizing medical papers. However, this new model can handle many different tasks using the same underlying technology. This versatility makes it a "generalist" model - and a powerful new tool in the hands of medical providers.

"BiomedGPT is based on foundation models, a recent development in AI," says Sun. "Foundation models are large, pre-trained AI systems that can be adapted to various tasks with minimal additional training. The generalist model described in the article has been trained on vast amounts of biomedical data, including images and text, enabling it to perform well across different applications."

"By evaluating 25 datasets across 9 biomedical tasks and different modalities," says Kai Zhang, a Lehigh PhD student advised by Sun who serves as first author of the Nature article, "BiomedGPT achieved 16 state-of-the-art results. A human evaluation of BiomedGPT on three radiology tasks showcased the model’s robust predictive abilities."

Zhang says that he is proud that the open-source codebase is available for other researchers to use as a springboard to drive further development and adoption.

The team reports that the technology behind BiomedGPT may one day help doctors by interpreting complex medical images, assist researchers by analyzing scientific literature, or even aid in drug discovery by predicting how molecules behave.

"The potential impact of such technology is significant," Zhang says, "as it could streamline many aspects of healthcare and research, making them faster and more accurate. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency."

A crucial step in the process was validation of the model's effectiveness and applicability in real-world healthcare settings.

"Clinical testing involves applying the AI model to real patient data to assess its accuracy, reliability, and safety," Sun says. "This testing ensures that the model performs well across different scenarios. The outcomes of these tests helped refine the model, demonstrating its potential to improve clinical decision-making and patient care."

Massachusetts General Hospital (MGH), a founding member of the Mass General Brigham healthcare system and teaching affiliate of Harvard Medical School, played a crucial role in the development and validation of the BiomedGPT model. The institution's involvement primarily focused on providing clinical expertise and facilitating the evaluation of the model's effectiveness in real-world healthcare settings. For instance, the model was tested with radiologists at MGH, where it demonstrated superior performance in tasks like visual question answering and radiology report generation. This collaboration helped ensure that the model was both accurate and practical for clinical use.

Other contributors to BiomedGPT include researchers from University of Georgia, Samsung Research America, University of Pennsylvania, Stanford University, University of Central Florida, UC-Santa Cruz, University of Texas-Health, Children’s Hospital of Philadelphia, and the Mayo Clinic.

"This research is highly interdisciplinary and collaborative," says Sun. "The research involves expertise from multiple fields, including computer science, medicine, radiology, and biomedical engineering. Each author contributes specialized knowledge necessary to develop, test, and validate the model across various biomedical tasks. Large-scale projects like this often require access to diverse datasets and computational resources, along with access to skills in algorithm development, model training, evaluation, and application to real-world scenarios, as well as clinical testing and validation.

"This was a true team effort," he says. "Creating something that can truly help the medical community improve patient outcomes across a wide range of issues is a very complex challenge. With such complexity, collaboration is key to creating impact through the application of science and engineering."

Zhang K, Zhou R, Adhikarla E, Yan Z, Liu Y, Yu J, Liu Z, Chen X, Davison BD, Ren H, Huang J, Chen C, Zhou Y, Fu S, Liu W, Liu T, Li X, Chen Y, He L, Zou J, Li Q, Liu H, Sun L.
A generalist vision-language foundation model for diverse biomedical tasks.
Nat Med. 2024 Aug 7. doi: 10.1038/s41591-024-03185-2

Most Popular Now

Philips Foundation 2024 Annual Report: E…

Marking its tenth anniversary, Philips Foundation released its 2024 Annual Report, highlighting a year in which the Philips Foundation helped provide access to quality healthcare for 46.5 million people around...

New AI Transforms Radiology with Speed, …

A first-of-its-kind generative AI system, developed in-house at Northwestern Medicine, is revolutionizing radiology - boosting productivity, identifying life-threatening conditions in milliseconds and offering a breakthrough solution to the global radiologist...

Scientists Argue for More FDA Oversight …

An agile, transparent, and ethics-driven oversight system is needed for the U.S. Food and Drug Administration (FDA) to balance innovation with patient safety when it comes to artificial intelligence-driven medical...

New Research Finds Specific Learning Str…

If data used to train artificial intelligence models for medical applications, such as hospitals across the Greater Toronto Area, differs from the real-world data, it could lead to patient harm...

Giving Doctors an AI-Powered Head Start …

Detection of melanoma and a range of other skin diseases will be faster and more accurate with a new artificial intelligence (AI) powered tool that analyses multiple imaging types simultaneously...

AI Agents for Oncology

Clinical decision-making in oncology is challenging and requires the analysis of various data types - from medical imaging and genetic information to patient records and treatment guidelines. To effectively support...

Patients say "Yes..ish" to the…

As artificial intelligence (AI) continues to be integrated in healthcare, a new multinational study involving Aarhus University sheds light on how dental patients really feel about its growing role in...

Brains vs. Bytes: Study Compares Diagnos…

A University of Maine study compared how well artificial intelligence (AI) models and human clinicians handled complex or sensitive medical cases. The study published in the Journal of Health Organization...

'AI Scientist' Suggests Combin…

An 'AI scientist', working in collaboration with human scientists, has found that combinations of cheap and safe drugs - used to treat conditions such as high cholesterol and alcohol dependence...

Start-ups in the Spotlight at MEDICA 202…

17 - 20 November 2025, Düsseldorf, Germany. MEDICA, the leading international trade fair and platform for healthcare innovations, will once again confirm its position as the world's number one hotspot for...