Nearly Half of FDA-Approved AI Devices Do Not Meet Clinical Validation Requirements
More than 43% of AI-powered medical devices approved by the US Food and Drug Administration (FDA) lacked published clinical validation data, a recent study by US multi-institutional researchers has revealed.
The study, published in Nature Medicine’s journal, analysed the clinical validation data of 521 AI-driven clinical devices approved by the FDA. According to the report, 226 of them had not been trained on real patient data.
What Does It mean To Be FDA-Approved?
The FDA is considered a reliable regulatory institution with significant influence over healthcare products in the US. When an AI device is approved by the FDA, it meets the organisation’s requirements for safety and effectiveness for medical use.
In recent years, the FDA has significantly increased its number of medical AI device approvals. Therefore, with AI increasingly penetrating the healthcare system, researchers from multiple US institutions including the University of North Carolina School of Medicine, Duke University and the University of Miami assessed the FDA’s current criteria for approving AI medical devices.
Clinical Validation Methods
The clinical validation of AI medical devices often relies on three main procedures. These are retrospective validation –which uses historical data-, prospective validation– based on real-time patient information-, and randomised controlled trials – prospective studies that use random assignment controls to maximise result accuracy-.
These procedures offer different degrees of scientific evidence, with prospective studies generally providing stronger validation. Randomised controlled trials are often considered the leading standard.
Concerning Results
To assess the reliability of the FDA’s authorisation process, the multi-institutional study examined the organisation’s official “Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices” database. Findings revealed that among the 521 FDA-approved devices, 27% had undergone retrospective validation, 28% had prospective validation, and only 4% -22 devices- had been tested through randomised controlled trials.
More concerningly, the study found that nearly half of the authorised devices –meaning 43% of the total- had no published clinical validation data at all. Some approved devices even relied on computer-generated images instead of real patient data, falling short of proper clinical validation standards.
In response to these results, Sammy Chouffani El Fassi, MD candidate at the UNC School of Medicine, research scholar at Duke Heart Center and leading author of the study, stated that “Although AI device manufacturers boast of the credibility of their technology with FDA authorisation, clearance does not mean that the devices have been properly evaluated for clinical effectiveness using real patient data”.
What Does This Mean for the Future of AI and Patient Care?
In recent years, AI tools have led to significant advancements in many fields of medicine including breast cancer and the diagnosis of obsessive-compulsive disorder among others. Therefore, regulatory entities like the FDA are constantly updating safety protocols to ensure responsible AI implementation in healthcare.
However, the lack of published clinical validation information for such a significant number of FDA-authorised AI medical devices has raised concerns about their real-world effectiveness. In response to this, Chouffani El Fassi affirmed the research team hoped their findings regarding FDA AI device approvals would have a positive impact on patient care at a large scale.
“We shared our findings with directors at the FDA who oversee medical device regulation, and we expect our work will inform their regulatory decision making. We also hope that our publication will inspire researchers and universities globally to conduct clinical validation studies on medical AI to improve the safety and effectiveness of these technologies.”
