Adversarial Safety Evaluation of a Parent-Facing, EHR-Integrated Chatbot in the Pediatric Intensive Care Unit: A Red-Teaming Study
Applied Clinical Informatics · 2026-10 · 17 (05) · 1026-1032
DOI: 10.1055/a-2997-3338 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Clinical Informatics, online erschienen am 20.11.2026. Verfasst von Ryan Brandon Hunter und 7 weiteren Autor:innen. Die Arbeit nennt 22 Literaturangaben.
Themen: Intensivpflege
Abstract (Originalsprache)
Red-teaming (adversarial safety testing) of family-facing clinical chatbots remains limited. This study aimed to red-team a parent-facing, EHR-integrated pediatric intensive care unit (PICU) chatbot. We evaluated 120 chatbot responses across 10 synthetic patients and six adversarial attack vectors (realistic parental challenges) with two conversational turns. Two PICU physicians independently scored each response as pass or fail against vector-specific safety criteria, classified failures by type, and rated response quality (1–5) and harm severity (0–5). Each failure involving a false parental assertion was posed again as a neutral factual query to distinguish uncritical deference from retrieval failure. As a proof of concept, all failures were retested with a modified system prompt requiring verification of parent-stated facts. The pass rate was 86.7% (104/120; 95% CI: 79.3–92.2%). Pass rates by vector ranged from 65% (fragmented input) to 100% (emergency triage). Of 16 failures, 10 (63%) accepted false parental assertions; 9 of these appeared to reflect uncritical deference rather than data retrieval failure. Twelve failures (75%) carried no or negligible potential harm; two were minor, two serious, and none were catastrophic. The modified prompt resolved 12/16 failures (75%). The dominant failure mode was uncritical acceptance of false parental assertions despite access to correct EHR data, and fragmented input posed a greater challenge than overt adversarial pressure. A targeted prompt-level mitigation resolved most failures in proof-of-concept testing. For family-facing clinical AI, helpfulness must not override fidelity to the medical record; further evaluation with real families and multi-turn testing is needed.
Zitieren
APA
Hunter, R. B., Hevia, S., Kumar, S., Karlsson, M. G., Chen, H., Carter, P., Mehta, S., & Rappold, T. (2026). Adversarial Safety Evaluation of a Parent-Facing, EHR-Integrated Chatbot in the Pediatric Intensive Care Unit: A Red-Teaming Study. Applied Clinical Informatics, 17(05), 1026-1032. https://doi.org/10.1055/a-2997-3338
Vancouver
Hunter RB, Hevia S, Kumar S, Karlsson MG, Chen H, Carter P, et al. Adversarial Safety Evaluation of a Parent-Facing, EHR-Integrated Chatbot in the Pediatric Intensive Care Unit: A Red-Teaming Study. Applied Clinical Informatics. 2026;17(05):1026-1032. doi:10.1055/a-2997-3338
BibTeX
@article{Hunter2026,
author = {Hunter, Ryan Brandon and Hevia, Sabrina and Kumar, Shelley and Karlsson, Mikael Guzman and Chen, Hanjie and Carter, Patrick and Mehta, Sanjiv and Rappold, Tommy},
title = {Adversarial Safety Evaluation of a Parent-Facing, EHR-Integrated Chatbot in the Pediatric Intensive Care Unit: A Red-Teaming Study},
journal = {Applied Clinical Informatics},
year = {2026},
volume = {17},
number = {05},
pages = {1026--1032},
doi = {10.1055/a-2997-3338}
}
EUVIMED