EUVIMEDEUVIMED entdecken
Crossref

Adversarial Safety Evaluation of a Parent-Facing, EHR-Integrated Chatbot in the Pediatric Intensive Care Unit: A Red-Teaming Study

Ryan Brandon Hunter, Sabrina Hevia, Shelley Kumar, Mikael Guzman Karlsson, Hanjie Chen, Patrick Carter, Sanjiv Mehta, Tommy Rappold

Applied Clinical Informatics · 2026-10 · 17 (05) · 1026-1032

DOI: 10.1055/a-2997-3338 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Clinical Informatics, online erschienen am 20.11.2026. Verfasst von Ryan Brandon Hunter und 7 weiteren Autor:innen. Die Arbeit nennt 22 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Red-teaming (adversarial safety testing) of family-facing clinical chatbots remains limited. This study aimed to red-team a parent-facing, EHR-integrated pediatric intensive care unit (PICU) chatbot. We evaluated 120 chatbot responses across 10 synthetic patients and six adversarial attack vectors (realistic parental challenges) with two conversational turns. Two PICU physicians independently scored each response as pass or fail against vector-specific safety criteria, classified failures by type, and rated response quality (1–5) and harm severity (0–5). Each failure involving a false parental assertion was posed again as a neutral factual query to distinguish uncritical deference from retrieval failure. As a proof of concept, all failures were retested with a modified system prompt requiring verification of parent-stated facts. The pass rate was 86.7% (104/120; 95% CI: 79.3–92.2%). Pass rates by vector ranged from 65% (fragmented input) to 100% (emergency triage). Of 16 failures, 10 (63%) accepted false parental assertions; 9 of these appeared to reflect uncritical deference rather than data retrieval failure. Twelve failures (75%) carried no or negligible potential harm; two were minor, two serious, and none were catastrophic. The modified prompt resolved 12/16 failures (75%). The dominant failure mode was uncritical acceptance of false parental assertions despite access to correct EHR data, and fragmented input posed a greater challenge than overt adversarial pressure. A targeted prompt-level mitigation resolved most failures in proof-of-concept testing. For family-facing clinical AI, helpfulness must not override fidelity to the medical record; further evaluation with real families and multi-turn testing is needed.

Zitieren

APA

Hunter, R. B., Hevia, S., Kumar, S., Karlsson, M. G., Chen, H., Carter, P., Mehta, S., & Rappold, T. (2026). Adversarial Safety Evaluation of a Parent-Facing, EHR-Integrated Chatbot in the Pediatric Intensive Care Unit: A Red-Teaming Study. Applied Clinical Informatics, 17(05), 1026-1032. https://doi.org/10.1055/a-2997-3338

Vancouver

Hunter RB, Hevia S, Kumar S, Karlsson MG, Chen H, Carter P, et al. Adversarial Safety Evaluation of a Parent-Facing, EHR-Integrated Chatbot in the Pediatric Intensive Care Unit: A Red-Teaming Study. Applied Clinical Informatics. 2026;17(05):1026-1032. doi:10.1055/a-2997-3338

BibTeX
@article{Hunter2026,
  author = {Hunter, Ryan Brandon and Hevia, Sabrina and Kumar, Shelley and Karlsson, Mikael Guzman and Chen, Hanjie and Carter, Patrick and Mehta, Sanjiv and Rappold, Tommy},
  title = {Adversarial Safety Evaluation of a Parent-Facing, EHR-Integrated Chatbot in the Pediatric Intensive Care Unit: A Red-Teaming Study},
  journal = {Applied Clinical Informatics},
  year = {2026},
  volume = {17},
  number = {05},
  pages = {1026--1032},
  doi = {10.1055/a-2997-3338}
}
Kostenlos starten →