Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations
PLOS ONE · 2025
DOI: 10.1371/journal.pone.0300487 ↗Kurz gefasst
Studienprotokoll in der Zeitschrift PLOS ONE, 2025. Verfasst von Edwin Kwan-Yeung Chiu und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong, China. Hauptthemen laut MEDLINE: Artificial Intelligence, Referral and Consultation. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Generative artificial intelligence (GenAI) has the potential to revolutionise healthcare delivery. The nuances of real-life clinical practice and complex clinical environments demand a rigorous, evidence-based approach to ensure safe and effective deployment of AI.
Methods
We present a protocol for the systematic evaluation of large language models (LLMs) as GenAI chatbots within the context of clinical microbiology and infectious diseases clinical consultations. We aim to critically assess recommendations produced by four leading GenAI models, including Claude 2, Gemini Pro, GPT-4.0, and a GPT-4.0-based custom AI chatbot.
Discussion
A standardised, healthcare-specific, universal prompt template is developed to elicit clinically impactful AI responses. Generated responses will be graded by two panels of practicing clinicians, encompassing a wide spectrum of domain expertise in clinical microbiology and virology, as well as infectious diseases. Evaluations will be performed using a 5-point Likert scale across four clinical domains: factual consistency, comprehensiveness, coherence, and medical harmfulness. Our study will offer insights into the feasibility, limitations, and boundaries of GenAI in clinical consultations, providing guidance for future research and clinical implementation. Ethical guidelines and safety guardrails should be developed to uphold patient safety and clinical standards.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong, China.
ORCID-Profile: Edwin Kwan-Yeung Chiu, Tom Wai-Hin Chung
Zitieren
APA
Chiu, E. K., & Chung, T. W. (2025). Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations. PLOS ONE. https://doi.org/10.1371/journal.pone.0300487
Vancouver
Chiu EK, Chung TW. Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations. PLOS ONE. 2025. doi:10.1371/journal.pone.0300487
BibTeX
@article{Chiu2025,
author = {Chiu, Edwin Kwan-Yeung and Chung, Tom Wai-Hin},
title = {Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations},
journal = {PLOS ONE},
year = {2025},
doi = {10.1371/journal.pone.0300487}
}
EUVIMED