EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques

Karolina Gaebe iD, Benjamin van der Woerd iD

PLOS One · 01.08.2025 · 20 (8) · e0325803

DOI: 10.1371/journal.pone.0325803 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 01.08.2025. Verfasst von Karolina Gaebe und 1 weiteren Autor:innen. Beteiligte Einrichtung: Division of Otolaryngology-Head and Neck Surgery, Department of Surgery - Michael G. DeGroote School of Medicine, Hamilton, Ontario, Canada. Hauptthemen laut MEDLINE: Natural Language Processing, Language. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) have demonstrated capabilities in natural language processing and critical reasoning. Studies investigating their potential use as healthcare diagnostic tools have largely relied on proprietary models like ChatGPT and have not explored the application of advanced prompt engineering techniques. This study aims to evaluate the diagnostic accuracy of three open-source LLMs and the role of prompt engineering using clinical scenarios.

Methods

We analyzed the performance of three open-source LLMs—llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768—using advanced prompt engineering when answering Medscape Clinical Challenge questions. Responses were recorded and evaluated for correctness, accuracy, precision, specificity, and sensitivity. A sensitivity analysis was conducted presenting the three LLMs with basic prompting challenge questions and excluding cases with visual assets. Results were compared with previously published performance data on GPT-3.5.

Results

Llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768 achieved correct responses in 79%, 65%, and 62% of cases, respectively, outperforming GPT-3.5 (74%). Diagnostic accuracy, precision, sensitivity, and specificity responses all outperformed those previously reported for GPT-3.5. Results generated using advanced prompting strategies were superior to those based on basic prompting. Sensitivity analysis revealed similar trends when cases with visual assets were excluded.

Discussion

Using advanced prompting techniques, LLMs can generate clinically accurate responses. The study highlights the limitations of proprietary models like ChatGPT, particularly in terms of accessibility and reproducibility due to version deprecation. Future research should employ prompt engineering techniques and prioritize the use of open-source models to ensure research replicability.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingLanguageHumansLarge Language Models

Autor:innen und Institutionen

  • Division of Otolaryngology-Head and Neck Surgery, Department of Surgery - Michael G. DeGroote School of Medicine, Hamilton, Ontario, Canada.

ORCID-Profile: Karolina Gaebe, Benjamin van der Woerd

Zitieren

APA

Gaebe, K., & van der Woerd, B. (2025). Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques. PLOS One, 20(8), e0325803. https://doi.org/10.1371/journal.pone.0325803

Vancouver

Gaebe K, van der Woerd B. Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques. PLOS One. 2025;20(8):e0325803. doi:10.1371/journal.pone.0325803

BibTeX
@article{Gaebe2025,
  author = {Gaebe, Karolina and van der Woerd, Benjamin},
  title = {Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques},
  journal = {PLOS One},
  year = {2025},
  volume = {20},
  number = {8},
  pages = {e0325803},
  doi = {10.1371/journal.pone.0325803}
}
Kostenlos starten →