EUVIMEDEUVIMED entdecken
StudienprotokollOpen Access · CC BY 4.0

Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations

Edwin Kwan-Yeung Chiu iD, Tom Wai-Hin Chung iD

PLOS ONE · 2025

DOI: 10.1371/journal.pone.0300487 ↗

Kurz gefasst

Studienprotokoll in der Zeitschrift PLOS ONE, 2025. Verfasst von Edwin Kwan-Yeung Chiu und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong, China. Hauptthemen laut MEDLINE: Artificial Intelligence, Referral and Consultation. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Generative artificial intelligence (GenAI) has the potential to revolutionise healthcare delivery. The nuances of real-life clinical practice and complex clinical environments demand a rigorous, evidence-based approach to ensure safe and effective deployment of AI.

Methods

We present a protocol for the systematic evaluation of large language models (LLMs) as GenAI chatbots within the context of clinical microbiology and infectious diseases clinical consultations. We aim to critically assess recommendations produced by four leading GenAI models, including Claude 2, Gemini Pro, GPT-4.0, and a GPT-4.0-based custom AI chatbot.

Discussion

A standardised, healthcare-specific, universal prompt template is developed to elicit clinically impactful AI responses. Generated responses will be graded by two panels of practicing clinicians, encompassing a wide spectrum of domain expertise in clinical microbiology and virology, as well as infectious diseases. Evaluations will be performed using a 5-point Likert scale across four clinical domains: factual consistency, comprehensiveness, coherence, and medical harmfulness. Our study will offer insights into the feasibility, limitations, and boundaries of GenAI in clinical consultations, providing guidance for future research and clinical implementation. Ethical guidelines and safety guardrails should be developed to uphold patient safety and clinical standards.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceReferral and ConsultationHumansDelivery of Health CareGenerative Artificial Intelligence

Autor:innen und Institutionen

  • Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong, China.

ORCID-Profile: Edwin Kwan-Yeung Chiu, Tom Wai-Hin Chung

Zitieren

APA

Chiu, E. K., & Chung, T. W. (2025). Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations. PLOS ONE. https://doi.org/10.1371/journal.pone.0300487

Vancouver

Chiu EK, Chung TW. Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations. PLOS ONE. 2025. doi:10.1371/journal.pone.0300487

BibTeX
@article{Chiu2025,
  author = {Chiu, Edwin Kwan-Yeung and Chung, Tom Wai-Hin},
  title = {Protocol for human evaluation of generative artificial intelligence chatbots in clinical consultations},
  journal = {PLOS ONE},
  year = {2025},
  doi = {10.1371/journal.pone.0300487}
}
Kostenlos starten →