Safety of human-AI cooperative decision-making within intensive care: A physical simulation study
Sicherheit kooperativer Mensch-KI-Entscheidungen auf Intensivstationen: physische Simulationsstudie
PLOS Digital Health · 24.02.2025 · 4 (2) · e0000726
DOI: 10.1371/journal.pdig.0000726 ↗Auf einen Blick
In einer physischen Simulation lehnten 92 % der Ärzt:innen unsichere KI-Empfehlungen ab, 29 % sichere. Unsichere Empfehlungen erhielten mehr Blickfixationen (+37 %).
Fragestellung
Wie sicher sind menschlich-KI-gestützte Therapieentscheidungen in einer Intensivstations-Simulation?
Methode
Physische Simulationsstudie in einer Intensivstation mit Ärzt:innen, Pflegekraft, Patient:innenpuppe, ICU-Datenchart und KI-Empfehlungssystem; Augentracking und Vorher-Nachher-Vergleich der Therapiepläne.
Ergebnisse
92 % der Clinicians lehnten unsichere KI-Empfehlungen ab, 29 % sichere. Unsichere Empfehlungen erhielten +37 % Blickfixationen. Auf KI-Erklärungen und klinische Informationen wurde nicht häufiger geschaut.
Schlussfolgerung der Autor:innen
Die Studie betont die Bedeutung menschlicher Aufsicht in sicherheitskritischen KI-Systemen und die Aussagekraft hochrealistischer Simulationen für die Bewertung von Mensch-KI-Systemen.
Kernaussagen
- 92 % lehnten unsichere KI-Empfehlungen ab
- Unsichere Empfehlungen: +37 % Blickfixationen
- KI-Erklärungen und klinische Infos erhielten keine höhere Aufmerksamkeit
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 24.02.2025. Verfasst von Paul Festor und 4 weiteren Autor:innen. Beteiligte Einrichtung: UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, United Kingdom. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The safety of Artificial Intelligence (AI) systems is as much one of human decision-making as a technological question. In AI-driven decision support systems, particularly in high-stakes settings such as healthcare, ensuring the safety of human-AI interactions is paramount, given the potential risks of following erroneous AI recommendations. To explore this question, we ran a safety-focused clinician-AI interaction study in a physical simulation suite. Physicians were placed in a simulated intensive care ward, with a human nurse (played by an experimenter), an ICU data chart, a high-fidelity patient mannequin and an AI recommender system on a display. Clinicians were asked to prescribe two drugs for the simulated patients suffering from sepsis and wore eye-tracking glasses to allow us to assess where their gaze was directed. We recorded clinician treatment plans before and after they saw the AI treatment recommendations, which could be either ‘safe’ or ‘unsafe’. 92% of clinicians rejected unsafe AI recommendations vs 29% of safe ones. Physicians paid increased attention (+37% gaze fixations) to unsafe AI recommendations vs safe ones. However, visual attention on AI explanations was not greater in unsafe scenarios. Similarly, clinical information (patient monitor, patient chart) did not receive more attention after an unsafe versus safe AI reveal suggesting that the physicians did not look back to these sources of information to investigate why the AI suggestion might be unsafe. Physicians were only successfully persuaded to change their dose by scripted comments from the bedside nurse 5% of the time. Our study emphasises the importance of human oversight in safety-critical AI and the value of evaluating human-AI systems in high-fidelity settings that more closely resemble real world practice.
Autor:innen und Institutionen
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, United Kingdom.
- Departments of Bioengineering and Computing, Brain and Behavior Lab, Imperial College London, London, United Kingdom.
- Division of Anaesthetics, Pain Medicine and Intensive Care, Imperial College London, London, United Kingdom.
- Digital Health and Data Science, Universität Bayreuth, Bayreuth, Germany.
ORCID-Profile: Aldo A Faisal
Zitieren
APA
Festor, P., Nagendran, M., Gordon, A. C., Faisal, A. A., & Komorowski, M. (2025). Safety of human-AI cooperative decision-making within intensive care: A physical simulation study. PLOS Digital Health, 4(2), e0000726. https://doi.org/10.1371/journal.pdig.0000726
Vancouver
Festor P, Nagendran M, Gordon AC, Faisal AA, Komorowski M. Safety of human-AI cooperative decision-making within intensive care: A physical simulation study. PLOS Digital Health. 2025;4(2):e0000726. doi:10.1371/journal.pdig.0000726
BibTeX
@article{Festor2025,
author = {Festor, Paul and Nagendran, Myura and Gordon, Anthony C and Faisal, Aldo A and Komorowski, Matthieu},
title = {Safety of human-AI cooperative decision-making within intensive care: A physical simulation study},
journal = {PLOS Digital Health},
year = {2025},
volume = {4},
number = {2},
pages = {e0000726},
doi = {10.1371/journal.pdig.0000726}
}
EUVIMED