EUVIMEDEUVIMED entdecken
Open Access

Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study

Christoph Raphael Buhr iD, Christopher Seifen iD, Katharina Bahr-Hamm iD, Tilman Huppertz iD, Johannes Pordzik iD, Harry Smith iD, Tom Kelsey iD, Andrew Blaikie iD, Christoph Matthias iD, Sebastian Kuhn iD, Jonas Eckrich iD

JMIR Formative Research · 2025

DOI: 10.2196/76896 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, 2025. Verfasst von Christoph Raphael Buhr und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Langenbeckstraße 1, Mainz, 55131, Germany, +…. Hauptthemen laut MEDLINE: Otolaryngology, Ambulatory Care, Programming Languages. Die Arbeit nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) have great potential to improve and make the work of clinicians more efficient. Previous studies have mainly focused on web-based services, such as ChatGPT, often with simulated cases. For the processing of personalized patient data, web-based services have major data protection concerns. Ensuring compliance with data protection and medical device regulations therefore remains a critical challenge for adopting LLMs in clinical settings.

Objective

This retrospective single-center study aimed to evaluate locally run LLMs (Gemma 2, Mistral Nemo, and Llama 3) in providing diagnosis and treatment recommendation for real-world outpatient cases in otorhinolaryngology (ORL).

Methods

Outpatient cases (n=30) from regular consultation hours and the emergency service at a university hospital ORL outpatient department were randomly selected. Documentation by ORL doctors, including anamnesis and examination results, was passed to the locally run LLMs (Gemma 2, Mistral Nemo, and Llama 3), which were asked to provide diagnostic and treatment strategies. Recommendations of the LLMs and the treating ORL doctors were rated by 3 experienced ORL consultants on a 6-point Likert scale for medical adequacy, conciseness, coherence, and comprehensibility. Moreover, consultants were asked whether the answers pose a risk to the patient’s safety. A modified Turing test was performed to distinguish responses generated by LLMs from those of doctors. Finally, the potential influence of the information generated by the LLMs on the raters’ own diagnosis and treatment opinions was evaluated.

Results

Over all categories, ORL doctors achieved superior ( P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

OtolaryngologyAmbulatory CareProgramming LanguagesRetrospective StudiesHumansFemaleMaleOutpatientsAdultMiddle AgedLarge Language Models

Stichwörter der Autor:innen

artificial intelligencechatbotdigital healthglobal healthlarge language modelslow- and middle-income countriesotorhinolaryngologytelehealthtelemedicine

Autor:innen und Institutionen

  • Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Langenbeckstraße 1, Mainz, 55131, Germany, +49 6131 17 7362.
  • School of Medicine, University of St Andrews, St Andrews, United Kingdom.
  • School of Computer Science, University of St Andrews, St Andrews, United Kingdom.
  • Institute for Digital Medicine, Philipps University Marburg, University Hospital Giessen and Marburg, Marburg, Germany.

ORCID-Profile: Christoph Raphael Buhr, Christopher Seifen, Katharina Bahr-Hamm, Tilman Huppertz, Johannes Pordzik, Harry Smith, Tom Kelsey, Andrew Blaikie, Christoph Matthias, Sebastian Kuhn, Jonas Eckrich

Zitieren

APA

Buhr, C. R., Seifen, C., Bahr-Hamm, K., Huppertz, T., Pordzik, J., Smith, H., Kelsey, T., Blaikie, A., Matthias, C., Kuhn, S., & Eckrich, J. (2025). Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study. JMIR Formative Research. https://doi.org/10.2196/76896

Vancouver

Buhr CR, Seifen C, Bahr-Hamm K, Huppertz T, Pordzik J, Smith H, et al. Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study. JMIR Formative Research. 2025. doi:10.2196/76896

BibTeX
@article{Buhr2025,
  author = {Buhr, Christoph Raphael and Seifen, Christopher and Bahr-Hamm, Katharina and Huppertz, Tilman and Pordzik, Johannes and Smith, Harry and Kelsey, Tom and Blaikie, Andrew and Matthias, Christoph and Kuhn, Sebastian and Eckrich, Jonas},
  title = {Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/76896}
}
Kostenlos starten →