EUVIMEDEUVIMED entdecken
Mixed-Methods-StudieOpen Access

Iterative Multidisciplinary Development and Evaluation of a Patient-Facing Social Determinants of Health Chatbot Using Synthetic Data Simulation: Mixed Methods Study

Iterative multidisziplinäre Entwicklung und Bewertung eines patientenorientierten Chatbots zu sozialen Gesundheitsdeterminanten (SDoH) mithilfe synthetischer Daten und Fallsimulation: Mixed-Methods-Studie

Anna M Maw iD, Alexander Lupi iD, Rachel Johnson-Koenke iD, Heather Coats iD, Li Zhou iD, Benjamin Li iD, James Mitchell iD, Alexander Kotz iD, Joseph Plasek iD, Foster Goss iD

JMIR Formative Research · 07.08.2026 · 10 · e89837-e89837

DOI: 10.2196/89837 ↗

Auf einen Blick

In 27 simulierten Fällen zeigte der SDoH-Chatbot hohe Bewertungen für Interpretation, Kommunikationsqualität, kulturelle Sensitivität und adaptive Fragen. Schwächer waren Domänenfokus, Vollständigkeit und Sicherheit.

Fragestellung

Beschreibt die Studie einen effizienten iterativen Ansatz zur Entwicklung und Bewertung eines patientenorientierten SDoH-Chatbots mit synthetischen Daten vor der klinischen Anwendung?

Methode

10-Kriterien-Bewertungsraster aus AI-Frameworks; Anwendung auf 27 synthetische klinische Szenarien; Rollenplay durch eine klinische Sozialarbeiterin; Bewertung durch 3 multidisziplinäre Expert:innen; quantitative und qualitative Analyse.

Ergebnisse

27 simulierte Fälle. Hohe positive Bewertungen für akkurate Interpretation (agreement=0,98, 95 %-KI 0,91–0,99), Kommunikationsqualität, kulturelle Sensitivität und adaptive Fragen. Niedriger bei Fokussierung/Vollständigkeit, Datenerfassung und Sicherheit.

Einschränkungen

Synthetische Fälle; keine externen Rater; keine Patientenbeteiligung; keine prospektive klinische Evaluation.

Schlussfolgerung der Autor:innen

Die Studie beschreibt einen formativen Machbarkeitsansatz zur iterativen Verfeinerung eines patientenorientierten SDoH-Chatbots und des Bewertungsrasters mit synthetischer Fallsimulation.

Kernaussagen

  • 27 simulierte Fälle mit multidisziplinärer Bewertung
  • Stärken bei Interpretation, Kommunikation, kultureller Sensitivität
  • Schwächen bei Vollständigkeit, Datenerfassung und Sicherheit

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Mixed-Methods-Studie in der Zeitschrift JMIR Formative Research, online erschienen am 07.08.2026. Verfasst von Anna M Maw und 9 weiteren Autor:innen. Beteiligte Einrichtung: Division of Hospital Medicine, University of Colorado School of Medicine, 12401 East 17th Avenue, Mailstop F-782, Aurora, CO, 80045, United States, 1…. Hauptthemen laut MEDLINE: Social Determinants of Health. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: AHRQ HHS.

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Systematic collection of social determinants of health (SDoH) data remains inconsistent across health care settings, despite its critical impact on patient outcomes. Large language model–powered chatbots offer promise for scalable SDoH data collection, but rigorous, feasible evaluation methods for patient-facing applications are lacking.

Objective

This study aimed to describe an efficient, iterative, multidisciplinary approach for developing and evaluating a patient-facing SDoH chatbot using synthetic data and case simulation, with the goal of optimizing both chatbot performance and the evaluation rubric prior to clinical deployment.

Methods

A 10-criterion evaluation rubric was adapted from established health care AI frameworks and applied to 27 synthetic clinical scenarios representing diverse SDoH profiles. Scenarios were role-played by a licensed clinical social worker, and chatbot-patient interactions were rated by 3 members of the research team that were multidisciplinary experts: a social worker, a nurse practitioner, and a physician. Quantitative analysis used percent agreement and Fleiss κ to characterize chatbot performance and rater consensus, with percent agreement selected due to the high prevalence of ceiling effects in several domains. Qualitative analysis synthesized rater feedback to guide iterative refinement of both chatbot prompts and rubric domains.

Results

Across 27 simulated cases, the chatbot received high proportions of positive ratings for accurate interpretation (agreement=0.98%, 95% CI 0.91‐0.99), communication quality, and cultural sensitivity (agreement=0.99%, 95% CI 0.93‐1.00), and appropriately adaptive questioning (agreement=0.99%, 95% CI 0.93‐1.00). Lower performance was observed in domain focus and completeness (agreement=0.51%, 95% CI 0.40‐0.61), completeness of data capture (agreement=0.59%, 95% CI 0.48‐0.69; Fleiss κ=0.18), and safety (agreement=0.69%, 95% CI 0.58‐0.78; Fleiss κ=−0.04), prompting targeted adaptations. Qualitative feedback highlighted the importance of distinguishing screening from clinical interviewing capabilities and informed the refinement of the rubric, including clarifying the definition of safety to focus on recognition of physical and mental health emergencies.

Conclusions

This study describes a formative feasibility approach for iterative refinement of a patient-facing SDoH chatbot and its evaluation rubric using synthetic case simulation. Future work will include independent external raters, patient stakeholders, repeated scenario testing, and prospective clinical evaluation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Social Determinants of HealthHumansLarge Language Models

Stichwörter der Autor:innen

AIchatbothealth care evaluationlarge language modelsrubricsocial determinants of healthsynthetic data

Autor:innen und Institutionen

  • Division of Hospital Medicine, University of Colorado School of Medicine, 12401 East 17th Avenue, Mailstop F-782, Aurora, CO, 80045, United States, 1 720 848 4289.
  • Adult and Child Consortium for Health Outcomes Research and Delivery Science (ACCORDS), University of Colorado, 1890 N Revere Ct, Aurora, CO, 80045, United States, 1 720 848 4289.
  • Department of Emergency Medicine, University of Colorado School of Medicine, Aurora, CO, United States.
  • Denver Health Medical Center, Denver, CO, United States.
  • College of Nursing, University of Colorado, Anschutz Medical Campus, Aurora, CO, United States.
  • Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States.

ORCID-Profile: Anna M Maw, Alexander Lupi, Rachel Johnson-Koenke, Heather Coats, Li Zhou, Benjamin Li, James Mitchell, Alexander Kotz, Joseph Plasek, Foster Goss

Förderung

  • AHRQ HHS

Zitieren

APA

Maw, A. M., Lupi, A., Johnson-Koenke, R., Coats, H., Zhou, L., Li, B., Mitchell, J., Kotz, A., Plasek, J., & Goss, F. (2026). Iterative Multidisciplinary Development and Evaluation of a Patient-Facing Social Determinants of Health Chatbot Using Synthetic Data Simulation: Mixed Methods Study. JMIR Formative Research, 10, e89837-e89837. https://doi.org/10.2196/89837

Vancouver

Maw AM, Lupi A, Johnson-Koenke R, Coats H, Zhou L, Li B, et al. Iterative Multidisciplinary Development and Evaluation of a Patient-Facing Social Determinants of Health Chatbot Using Synthetic Data Simulation: Mixed Methods Study. JMIR Formative Research. 2026;10:e89837-e89837. doi:10.2196/89837

BibTeX
@article{Maw2026,
  author = {Maw, Anna M and Lupi, Alexander and Johnson-Koenke, Rachel and Coats, Heather and Zhou, Li and Li, Benjamin and Mitchell, James and Kotz, Alexander and Plasek, Joseph and Goss, Foster},
  title = {Iterative Multidisciplinary Development and Evaluation of a Patient-Facing Social Determinants of Health Chatbot Using Synthetic Data Simulation: Mixed Methods Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e89837--e89837},
  doi = {10.2196/89837}
}
Kostenlos starten →