EUVIMEDEUVIMED entdecken
ÜbersichtsarbeitSystematische ÜbersichtOpen Access

Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review

Yining Hua iD, Winna Xia iD, David Bates iD, George Luke Hartstein iD, Hyungjin Tom Kim iD, Michael Li iD, Benjamin W Nelson iD, Charles Stromeyer Iv iD, Darlene King iD, Jina Suh iD, Li Zhou iD, John Torous iD

JMIR AI · 07.11.2025 · 4 · e69006

DOI: 10.2196/69006 ↗

Kurz gefasst

Übersichtsarbeit in der Zeitschrift JMIR AI, online erschienen am 07.11.2025. Verfasst von Yining Hua und 11 weiteren Autor:innen. Beteiligte Einrichtung: Technology and Operations Management, Harvard Business School, Cambridge, MA, United States. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Health care chatbots are rapidly proliferating, while generative artificial intelligence (AI) outpaces existing evaluation standards.

Objective

We aimed to develop a structured, stakeholder-informed framework to standardize evaluation of health care chatbots.

Methods

PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)–guided searches across multiple databases identified 266 records; 152 were screened, 21 full texts were assessed, and 11 frameworks were included. We extracted 356 questions (refined to 271 by deduplication and relevance review), mapped items to Coalition for Health AI constructs, and organized them with iterative input from clinicians, patients, developers, epidemiologists, and policymakers.

Results

We developed the Health Care AI Chatbot Evaluation Framework (HAICEF), a hierarchical framework with 3 priority domains (safety, privacy, and fairness; trustworthiness and usefulness; and design and operational effectiveness) and 18 second-level and 60 third-level constructs covering 271 questions. Emphasis includes data provenance and harm control; Health Insurance Portability and Accountability Act/General Data Protection Regulation–aligned privacy and security; bias management; and reliability, transparency, and workflow integration. Question distribution across domains is as follows: design and operational effectiveness, 40%; trustworthiness and usefulness, 39%; and safety, privacy and fairness, 21%. The framework accommodates both patient-facing and back-office use cases.

Conclusions

HAICEF provides an adaptable scaffold for standardized evaluation and responsible implementation of health care chatbots. Planned next steps include prospective validation across settings and a Delphi consensus to extend accountability and accessibility assurances.

Schlagwörter

Stichwörter der Autor:innen

AI chatbotsartificial intelligenceevaluation frameworkgenerative AIlarge language model

Autor:innen und Institutionen

  • Technology and Operations Management, Harvard Business School, Cambridge, MA, United States.
  • Department of Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA, United States.
  • Division of Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States.
  • Department of Psychiatry, Thomas Jefferson University, Philadelphia, PA, United States.
  • Department of Psychiatry and Human Behavior, Brown University, Providence, RI, United States.
  • Department of Psychiatry, University of Texas Southwestern, Houston, TX, United States.

ORCID-Profile: Yining Hua, Winna Xia, David Bates, George Luke Hartstein, Hyungjin Tom Kim, Michael Li, Benjamin W Nelson, Charles Stromeyer Iv, Darlene King, Jina Suh, Li Zhou, John Torous

Zitieren

APA

Hua, Y., Xia, W., Bates, D., Hartstein, G. L., Kim, H. T., Li, M., Nelson, B. W., Stromeyer Iv, C., King, D., Suh, J., Zhou, L., & Torous, J. (2025). Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review. JMIR AI, 4, e69006. https://doi.org/10.2196/69006

Vancouver

Hua Y, Xia W, Bates D, Hartstein GL, Kim HT, Li M, et al. Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review. JMIR AI. 2025;4:e69006. doi:10.2196/69006

BibTeX
@article{Hua2025,
  author = {Hua, Yining and Xia, Winna and Bates, David and Hartstein, George Luke and Kim, Hyungjin Tom and Li, Michael and Nelson, Benjamin W and Stromeyer Iv, Charles and King, Darlene and Suh, Jina and Zhou, Li and Torous, John},
  title = {Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review},
  journal = {JMIR AI},
  year = {2025},
  volume = {4},
  pages = {e69006},
  doi = {10.2196/69006}
}
Kostenlos starten →