EUVIMEDOpdag EUVIMED
Open access

A Patient Simulation Framework for Risk Assessment of Conversational Health Care AI: Development and Evaluation Study

Ein Patienten-Simulationsframework zur Risikobewertung von konversationeller Gesundheits-KI: Entwicklungs- und Evaluationsstudie

Md Tanvir Rouf Shawon, Mohammad Sabik Irbaz, Hadeel R A Elyazori, Keerti Reddy Resapu, Yili Lin, Vladimir Franzuela Cardenas, K Pierre Eklou, Farrokh Alemi, Kevin Lybarger

JMIR AI · 2026-10-05 · 5 · e100772-e100772

DOI: 10.2196/100772 ↗

Auf einen Blick

Ein Simulationsframework nach NIST AI RMF zeigte Leistungsrisiken einer konversationellen Entscheidungsunterstützung für die Auswahl von Antidepressiva. Die Leistung sank mit dem Gesundheitskompetenz-Gradienten.

Fragestellung

Entwicklung und Validierung eines Patienten-Simulationsframeworks zur Identifikation von Leistungsrisiken konversationeller klinischer KI über medizinische, sprachliche und Verhaltensvariationen hinweg.

Methode

Framework mit 3 Profilachsen: medizinisch, sprachlich und behaviorale Profile; 500 simulierte Gespräche; Bewertung von Profiltreue, Konzeptabruf und Empfehlungen mit menschlicher Annotation und LLM-Judge.

Ergebnisse

Medizinische Konzepte mit 96,6 % Genauigkeit über 8210 Konzepte; Interrater-Übereinstimmung κ=0,73; LLM vs. Mensch κ=0,78; Verhalten κ=0,93; Sprache κ=0,61. Rank-1-Konzeptabruf: 47,6 % vs. 81,9 %.

Einschränkungen

Nur auf die Auswahl von Antidepressiva angewendet; Übertragbarkeit auf andere klinische Entscheidungshilfen bleibt offen.

Schlussfolgerung der Autor:innen

Patientensimulation auf Basis des NIST AI RMF deckt messbare Leistungsrisiken konversationeller Gesundheits-KI auf, die statische Benchmarks übersehen. Gesundheitskompetenz wirkt als struktureller Risikofaktor.

Kernaussagen

  • 96,6 % Genauigkeit bei medizinischen Konzepten im Simulator
  • Leistung nahm mit höherer Gesundheitskompetenz zu
  • Nur Antidepressiva-Auswahl untersucht; andere Aufgaben offen

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Journal article in JMIR AI, published online on 2026-10-05. Written by Md Tanvir Rouf Shawon and 8 co-authors. Institution: Department of Computer Science, College of Engineering and Computing, George Mason University, Nguyen Engineering Building, 4511 Patriot Cir, Fairfax…. 106 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Conversational AI systems are increasingly being deployed in health care for clinical decision support, but their performance varies substantially across patient communication styles, health literacy levels, and behavioral patterns. Static benchmarks cannot capture multiturn dynamics through which this variation compounds, and no current evaluation framework implements structured AI risk management guidance for conversational health care AI. The result is a structural risk: AI systems may perform well in aggregate while failing disproportionately for the populations they are intended to help.

Objective

This study aimed to develop and validate a patient simulation framework that aligns with the National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF) Map and Measure functions, providing an empirical basis for identifying and characterizing performance risks in conversational clinical AI across medical, linguistic, and behavioral patient variations. We applied the framework to a conversational decision aid for antidepressant selection in major depressive disorder (the AI decision aid).

Methods

The simulator integrated three profile dimensions: (1) medical profiles constructed from All of Us electronic health records using risk ratio gating; (2) linguistic profiles modeling a health literacy gradient and condition-specific communication; and (3) behavioral profiles representing cooperative, distracted, and adversarial engagement. We generated 500 simulated conversations and evaluated profile fidelity through human annotation and a large language model (LLM) judge, and then assessed downstream effects on the AI decision aid’s concept retrieval and antidepressant recommendations.

Results

The patient simulator expressed medical concepts with high fidelity (96.6% accuracy across 8210 concepts), with substantial human interannotator agreement (κ=0.73) and LLM-judge agreement against human annotators (κ=0.78). Behavioral profiles were reliably distinguished (κ=0.93; near perfect agreement), and linguistic profiles showed substantial agreement at the lower bound of the substantial range (κ=0.61), which can be considered adequate to support profile-level analysis. The framework revealed monotonic degradation in AI decision aid performance across the health literacy gradient. Rank-1 concept retrieval increased from 47.6% for limited health literacy to 81.9% for proficient health literacy, with corresponding declines in antidepressant recommendation accuracy.

Conclusions

Patient simulation grounded in the NIST AI RMF exposes measurable performance risks in conversational health care AI that static benchmarks miss, with direct equity implications. Health literacy operates as a structural risk factor, with degraded performance concentrated in patients carrying the greatest burden of psychiatric illness. The framework supports targeted risk-mitigation interventions before deployment. While we evaluated the framework only on antidepressant selection, extending it to other clinical decision-aid tasks remains an assignment for future work.

Subjects

Author keywords

AI risk managementconsumer healthconversational agentsdepression managementhealth informaticslarge language modelsnatural language generationpatient simulationtrustworthy AI

Authors and institutions

  • Department of Computer Science, College of Engineering and Computing, George Mason University, Nguyen Engineering Building, 4511 Patriot Cir, Fairfax, VA, 22032, United States, 1 571-663-5183.
  • Department of Information Sciences and Technology, College of Engineering and Computing, George Mason University, Fairfax, VA, United States.
  • Department of Health Administration and Policy, College of Public Health, George Mason University, Fairfax, VA, United States.
  • School of Nursing, College of Public Health, George Mason University, Fairfax, VA, United States.

Cite

APA

Shawon, M. T. R., Irbaz, M. S., Elyazori, H. R. A., Resapu, K. R., Lin, Y., Cardenas, V. F., Eklou, K. P., Alemi, F., & Lybarger, K. (2026). A Patient Simulation Framework for Risk Assessment of Conversational Health Care AI: Development and Evaluation Study. JMIR AI, 5, e100772-e100772. https://doi.org/10.2196/100772

Vancouver

Shawon MTR, Irbaz MS, Elyazori HRA, Resapu KR, Lin Y, Cardenas VF, et al. A Patient Simulation Framework for Risk Assessment of Conversational Health Care AI: Development and Evaluation Study. JMIR AI. 2026;5:e100772-e100772. doi:10.2196/100772

BibTeX
@article{Shawon2026,
  author = {Shawon, Md Tanvir Rouf and Irbaz, Mohammad Sabik and Elyazori, Hadeel R A and Resapu, Keerti Reddy and Lin, Yili and Cardenas, Vladimir Franzuela and Eklou, K Pierre and Alemi, Farrokh and Lybarger, Kevin},
  title = {A Patient Simulation Framework for Risk Assessment of Conversational Health Care AI: Development and Evaluation Study},
  journal = {JMIR AI},
  year = {2026},
  volume = {5},
  pages = {e100772--e100772},
  doi = {10.2196/100772}
}
Start for free →