Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study
JMIR Formative Research · 2025
DOI: 10.2196/76896 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Christoph Raphael Buhr und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Langenbeckstraße 1, Mainz, 55131, Germany, +…. Hauptthemen laut MEDLINE: Otolaryngology, Ambulatory Care, Programming Languages. Die Arbeit nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) have great potential to improve and make the work of clinicians more efficient. Previous studies have mainly focused on web-based services, such as ChatGPT, often with simulated cases. For the processing of personalized patient data, web-based services have major data protection concerns. Ensuring compliance with data protection and medical device regulations therefore remains a critical challenge for adopting LLMs in clinical settings.
Objective
This retrospective single-center study aimed to evaluate locally run LLMs (Gemma 2, Mistral Nemo, and Llama 3) in providing diagnosis and treatment recommendation for real-world outpatient cases in otorhinolaryngology (ORL).
Methods
Outpatient cases (n=30) from regular consultation hours and the emergency service at a university hospital ORL outpatient department were randomly selected. Documentation by ORL doctors, including anamnesis and examination results, was passed to the locally run LLMs (Gemma 2, Mistral Nemo, and Llama 3), which were asked to provide diagnostic and treatment strategies. Recommendations of the LLMs and the treating ORL doctors were rated by 3 experienced ORL consultants on a 6-point Likert scale for medical adequacy, conciseness, coherence, and comprehensibility. Moreover, consultants were asked whether the answers pose a risk to the patient’s safety. A modified Turing test was performed to distinguish responses generated by LLMs from those of doctors. Finally, the potential influence of the information generated by the LLMs on the raters’ own diagnosis and treatment opinions was evaluated.
Results
Over all categories, ORL doctors achieved superior ( P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Langenbeckstraße 1, Mainz, 55131, Germany, +49 6131 17 7362.
- School of Medicine, University of St Andrews, St Andrews, United Kingdom.
- School of Computer Science, University of St Andrews, St Andrews, United Kingdom.
- Institute for Digital Medicine, Philipps University Marburg, University Hospital Giessen and Marburg, Marburg, Germany.
ORCID-Profile: Christoph Raphael Buhr, Christopher Seifen, Katharina Bahr-Hamm, Tilman Huppertz, Johannes Pordzik, Harry Smith, Tom Kelsey, Andrew Blaikie, Christoph Matthias, Sebastian Kuhn, Jonas Eckrich
Zitieren
APA
Buhr, C. R., Seifen, C., Bahr-Hamm, K., Huppertz, T., Pordzik, J., Smith, H., Kelsey, T., Blaikie, A., Matthias, C., Kuhn, S., & Eckrich, J. (2025). Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study. JMIR Formative Research. https://doi.org/10.2196/76896
Vancouver
Buhr CR, Seifen C, Bahr-Hamm K, Huppertz T, Pordzik J, Smith H, et al. Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study. JMIR Formative Research. 2025. doi:10.2196/76896
BibTeX
@article{Buhr2025,
author = {Buhr, Christoph Raphael and Seifen, Christopher and Bahr-Hamm, Katharina and Huppertz, Tilman and Pordzik, Johannes and Smith, Harry and Kelsey, Tom and Blaikie, Andrew and Matthias, Christoph and Kuhn, Sebastian and Eckrich, Jonas},
title = {Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study},
journal = {JMIR Formative Research},
year = {2025},
doi = {10.2196/76896}
}
EUVIMED