News | Artificial Intelligence | October 04, 2023

Study highlights risk of using foundation models in medical imaging AI

Study highlights risk of using foundation models in medical imaging AI

Ben Glocker, PhD 


October 4, 2023 — An AI chest X-ray foundation model for disease detection demonstrated racial and sex-related bias leading to uneven performance across patient subgroups and may be unsafe for clinical applications, according to a study published today in Radiology: Artificial Intelligence, a journal of the Radiological Society of North America (RSNA). The study aims to highlight the potential risks for using foundation models in the development of medical imaging artificial intelligence. 

“There’s been a lot of work developing AI models to help doctors detect disease in medical scans,” said lead researcher Ben Glocker, Ph.D., professor of machine learning for imaging at Imperial College London in the U.K. “However, it can be quite difficult to get enough training data for a specific disease that is representative of all patient groups.” 

Due to the difficulty of collecting large volumes of high-quality training data, the AI field has moved toward using deep-learning foundation models that have been trained for other purposes. Foundation models are AI neural networks that have been trained on large, often unlabeled datasets which handle jobs from translating text to analyzing medical images. 

“Despite their increasing popularity, we know little about potential biases in foundation models that could affect downstream uses,” Dr. Glocker said.   

Dr. Glocker’s research team compared the performance of a recently published chest X-ray foundation model and a reference model built by the team in evaluating 127,118 chest X-rays with associated diagnostic labels. The pre-trained foundation model was built with more than 800,000 chest X-rays from India and the U.S. 

The researchers completed a comprehensive performance analysis to determine how well the models performed for individual subgroups. The 42,884 patients (mean age, 63; 23,623 male) in the study group included Asian, Black and white patients. 

Bias analysis showed significant differences between features related to disease detection across biological sex and race. 

“Our bias analysis showed that the foundation model consistently underperformed compared to the reference model,” Dr. Glocker said. “We observed a decline in disease classification performance and specific disparities in protected subgroups.” 

Significant differences were found between male and female and Asian and Black patients in the features related to disease detection. Compared with the average model performance across all subgroups, classification performance on the ‘no finding’ label dropped between 6.8% and 7.8% for female patients, and performance in detecting ‘pleural effusion’—a buildup of fluid around the lungs—dropped between 10.7% and 11.6% for Black patients. 

“Dataset size alone does not guarantee a better or fairer model,” Dr. Glocker said. “We need to be very careful about data collection to ensure diversity and representativeness.” 

He noted that it’s important that foundation models are published and shared. 

“To minimize the risk of bias associated with the use of foundation models for clinical decision-making, these models need to be fully accessible and transparent,” he said. 

Dr. Glocker is an advocate for comprehensive bias analysis as an integral part of the development and auditing of foundation models. 

“AI is often seen as a black box, but that’s not entirely true,” he said. “We can open the box and inspect the features. Model inspection is one way of continuously monitoring and flagging issues that need a second look.” 

The work doesn’t start with the AI model, it starts with the data used to build it, Dr. Glocker noted. 

“As we collect the next dataset, we need to, from day one, make sure AI is being used in a way that will benefit everyone,” he said. 

For more information: www.rsna.org


Related Content

News | Breast Imaging

Aug. 28, 2024 — Rezolut, LLC recently debuted its latest offering for patients during their annual mammogram ...

Time August 29, 2024
arrow
News | Digital Pathology

Paige has launched OmniScreen, an AI-driven biomarker module capable of evaluating over 505 genes and detecting 1,228 ...

Time August 27, 2024
arrow
News | Computed Tomography (CT)

SPONSORED CONTENT — Fujifilm’s latest CT technology brings exceptional image quality to a compact and user- and patient ...

Time August 06, 2024
arrow
News | RSNA

July 31, 2024 — The National Imaging Informatics Course (NIIC), a pioneering program in the radiology field, will return ...

Time July 31, 2024
arrow
News | Radiology Business

July 31, 2024 — The American Registry of Radiologic Technologists (ARRT) announced the three Registered Technologists (R ...

Time July 31, 2024
arrow
Feature | Radiation Oncology | By Christine Book

News emerging from several leading organizations and vendors in the radiation therapy arena came in at a fast pace in ...

Time July 30, 2024
arrow
Feature | Computed Tomography (CT) | By Melinda Taschetta-Millane

In the ever-evolving landscape of medical imaging, computed tomography (CT) stands out as a cornerstone technology ...

Time July 30, 2024
arrow
Videos | Radiology Business

Find actionable insights to achieve sustainability and savings in radiology in this newest of ITN’s “One on One” video ...

Time July 30, 2024
arrow
News | Breast Imaging

July 29, 2024 — Lunit, a leading provider of AI-powered solutions for cancer diagnostics and therapeutics, announced the ...

Time July 29, 2024
arrow
News | Breast Imaging

July 29, 2024 — iCAD, Inc., a global leader in clinically proven AI-powered cancer detection solutions, announced a ...

Time July 29, 2024
arrow
Subscribe Now