Ethics of AI in Scientific Research: Case Study on AI-Powered Diagnostic Tools
Introduction:
The following case study was developed as part of independent research on the ethical use of artificial intelligence in science, supported by the Hackworth Applied Ethics Research Grant through the Markkula Center for Applied Ethics at Santa Clara University. It is intended to prompt critical discussion among scientists, students, and faculty navigating the growing role of AI in scientific research and clinical settings.
Case Study:
A healthcare technology company has developed an AI diagnostic tool that is intended to be used by hospitals to detect early biomarkers of a neurodegenerative disease. This new technology can identify markers years before a patient would display symptoms. Real-world technologies demonstrate this potential, such as an AI-powered blood test developed by researchers that achieved an accuracy of 92.3% and outperformed traditional diagnostic approaches (Source).
Similarly, this fictional new biomarker detection tool outperforms physicians on accuracy. However, the technology is considered a “black box,” meaning the internal mechanisms that inform the tool’s reasoning are unclear, even to the developers themselves. This is an ongoing concern for complex AI models used in healthcare (Source). Specifically, this tool’s outputs include a tentative diagnosis and a confidence score but no explanation of the reasoning. The healthcare technology company defends this tool, arguing that its accuracy and early detection save lives that would otherwise be lost due to late diagnosis. On the other hand, critics argue that diagnoses without clear explanations or reasoning undermine patient autonomy and could lead to clinicians deferring to AI judgment rather than their own assessments. Critics are also skeptical of the studies behind these technologies, as it is unknown what data was used to calibrate these tools, leaving room for bias toward specific genders, ethnicities, or other forms of identity.
Discussion Questions:
What role does consent of the patient play when implementing this technology?
If this AI technology misdiagnoses a user, who should be considered responsible?
How should accuracy and transparency be weighed against each other?
What kind of solutions could help reduce bias in this situation? For example, should doctors make assessments before or after viewing this AI tool’s output?
What questions should be asked regarding the diversity of datasets used to train this AI tool? How might dataset bias affect accuracy across different patients?