Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques
PLOS One · 01.08.2025 · 20 (8) · e0325803
DOI: 10.1371/journal.pone.0325803 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS One, online erschienen am 01.08.2025. Verfasst von Karolina Gaebe und 1 weiteren Autor:innen. Beteiligte Einrichtung: Division of Otolaryngology-Head and Neck Surgery, Department of Surgery - Michael G. DeGroote School of Medicine, Hamilton, Ontario, Canada. Hauptthemen laut MEDLINE: Natural Language Processing, Language. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Large language models (LLMs) have demonstrated capabilities in natural language processing and critical reasoning. Studies investigating their potential use as healthcare diagnostic tools have largely relied on proprietary models like ChatGPT and have not explored the application of advanced prompt engineering techniques. This study aims to evaluate the diagnostic accuracy of three open-source LLMs and the role of prompt engineering using clinical scenarios.
Methods
We analyzed the performance of three open-source LLMs—llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768—using advanced prompt engineering when answering Medscape Clinical Challenge questions. Responses were recorded and evaluated for correctness, accuracy, precision, specificity, and sensitivity. A sensitivity analysis was conducted presenting the three LLMs with basic prompting challenge questions and excluding cases with visual assets. Results were compared with previously published performance data on GPT-3.5.
Results
Llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768 achieved correct responses in 79%, 65%, and 62% of cases, respectively, outperforming GPT-3.5 (74%). Diagnostic accuracy, precision, sensitivity, and specificity responses all outperformed those previously reported for GPT-3.5. Results generated using advanced prompting strategies were superior to those based on basic prompting. Sensitivity analysis revealed similar trends when cases with visual assets were excluded.
Discussion
Using advanced prompting techniques, LLMs can generate clinically accurate responses. The study highlights the limitations of proprietary models like ChatGPT, particularly in terms of accessibility and reproducibility due to version deprecation. Future research should employ prompt engineering techniques and prioritize the use of open-source models to ensure research replicability.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Division of Otolaryngology-Head and Neck Surgery, Department of Surgery - Michael G. DeGroote School of Medicine, Hamilton, Ontario, Canada.
ORCID-Profile: Karolina Gaebe, Benjamin van der Woerd
Zitieren
APA
Gaebe, K., & van der Woerd, B. (2025). Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques. PLOS One, 20(8), e0325803. https://doi.org/10.1371/journal.pone.0325803
Vancouver
Gaebe K, van der Woerd B. Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques. PLOS One. 2025;20(8):e0325803. doi:10.1371/journal.pone.0325803
BibTeX
@article{Gaebe2025,
author = {Gaebe, Karolina and van der Woerd, Benjamin},
title = {Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques},
journal = {PLOS One},
year = {2025},
volume = {20},
number = {8},
pages = {e0325803},
doi = {10.1371/journal.pone.0325803}
}
EUVIMED