Historical Evolution and Medical Device Approval
In the 90s, the FDA was already taking its first steps in regulating medical devices that used artificial intelligence. A pioneering example is PAPNET, a system that used neural networks to assist in the diagnosis of cervical cancer in women. This milestone opened the door to the incorporation of algorithms in medical diagnosis.
Today, it is estimated that the FDA has approved around 1,000 AI-based medical devices, many of which are designed to support clinical decisions in areas such as radiology, cardiology, and ophthalmology. However, simply having regulatory approval does not guarantee successful integration into daily practice.
The Importance of Explainability in AI
One of the main obstacles to the full adoption of AI in healthcare is the problem known as the "black box." Many algorithms offer accurate results in retrospective studies, but their internal functioning is difficult for clinical professionals to interpret.
This lack of interpretability is especially critical in environments where decision-making must be based on clear explanations. For example, the IDx-DR device, approved for diabetic retinopathy detection, offers a binary result (positive or negative) without providing details on how that conclusion was reached. Although its clinical studies have shown acceptable accuracy, the absence of a detailed explanation has generated skepticism among some physicians, who prefer to have additional information to support their clinical judgment.
Concrete Cases of Lack of Trust
1. Limited Validation and Overfitting
Numerous approved devices have been validated primarily with historical data and in a single clinical center. A recent study showed that, out of 141 evaluated algorithms, most had only retrospective validation, and very few had undergone prospective studies or randomized controlled trials. This limitation increases the risk of overfitting and population biases, which can cause the algorithm's performance to decline when applied in clinical environments different from those in which it was trained.
2. Algorithms with Inherent Biases
A case that captured media attention was that of an algorithm used to allocate resources in healthcare systems in the United States. This algorithm, based on historical data of medical service usage, tended to assign less complex treatments to African American patients, erroneously concluding that their conditions were less severe. The root of the problem lay in the fact that the data reflected existing inequalities in access to care, which resulted in discriminatory decisions and, consequently, a significant loss of trust in these automated systems.
3. False Alerts in Sepsis Detection Systems
Another example is algorithms designed to predict sepsis. In some systems, a high rate of false alerts has been observed, which generates fatigue in clinical staff. When doctors receive too many alerts that turn out to be inaccurate, the tendency is to ignore them, which can have serious consequences for patient safety. The inability to explain and adjust these alerts in real-time contributes to a perception of ineffectiveness and distrust in the technology.
The FDA's Perspective and the Need for More Rigorous Standards
The U.S. Food and Drug Administration (FDA) recognizes these challenges and is working to establish regulatory frameworks that encourage innovation without sacrificing safety and efficacy. In a recent article published in JAMA, senior FDA officials emphasized the importance of developing an adaptable, science-based regulatory scheme that allows for continuous assessment of AI performance in clinical practice.
- Adaptability in validation: Encourage prospective studies and controlled trials that go beyond retrospective validation.
- Transparency in decision-making: Promote the development of explainable AI techniques, such as heat maps or saliency maps, that allow physicians to better understand the algorithm's decision-making process.
- AI lifecycle management: Implement continuous monitoring and supervision mechanisms that ensure algorithms maintain their effectiveness in real clinical environments.
Towards a Future with Explainable AI
The integration of AI in healthcare promises significant improvements in diagnostic accuracy, workflow optimization, and treatment personalization. However, for these benefits to fully materialize, it is essential to advance in the field of explainable AI. Developers, researchers, and regulators must work closely together to create algorithms that are not only accurate but also transparent and adaptable to different clinical contexts.
Conclusion
The history of AI in healthcare shows remarkable progress, from the first devices in the 90s to the present, with nearly 1,000 devices approved by the FDA. However, the lack of explainability, limited validations, and concrete cases of bias have generated significant distrust in the medical community. The solution lies in promoting the development of explainable AI and establishing regulatory frameworks that ensure the transparency and effectiveness of these systems in real clinical practice.
The evolution towards an AI that can explain its decisions will not only increase professionals' trust but will also allow the technology to be safely and responsibly integrated into healthcare, saving lives and improving patient outcomes.