EUVIMEDObjevte EUVIMED
Open access

Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study

Christoph Raphael Buhr iD, Christopher Seifen iD, Katharina Bahr-Hamm iD, Tilman Huppertz iD, Johannes Pordzik iD, Harry Smith iD, Tom Kelsey iD, Andrew Blaikie iD, Christoph Matthias iD, Sebastian Kuhn iD, Jonas Eckrich iD

JMIR Formative Research · 2025

DOI: 10.2196/76896 ↗

At a glance

Journal article in JMIR Formative Research, 2025. Written by Christoph Raphael Buhr and 10 co-authors. Institution: Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Langenbeckstraße 1, Mainz, 55131, Germany, +…. Major MEDLINE topics: Otolaryngology, Ambulatory Care, Programming Languages. 19 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Large language models (LLMs) have great potential to improve and make the work of clinicians more efficient. Previous studies have mainly focused on web-based services, such as ChatGPT, often with simulated cases. For the processing of personalized patient data, web-based services have major data protection concerns. Ensuring compliance with data protection and medical device regulations therefore remains a critical challenge for adopting LLMs in clinical settings.

Objective

This retrospective single-center study aimed to evaluate locally run LLMs (Gemma 2, Mistral Nemo, and Llama 3) in providing diagnosis and treatment recommendation for real-world outpatient cases in otorhinolaryngology (ORL).

Methods

Outpatient cases (n=30) from regular consultation hours and the emergency service at a university hospital ORL outpatient department were randomly selected. Documentation by ORL doctors, including anamnesis and examination results, was passed to the locally run LLMs (Gemma 2, Mistral Nemo, and Llama 3), which were asked to provide diagnostic and treatment strategies. Recommendations of the LLMs and the treating ORL doctors were rated by 3 experienced ORL consultants on a 6-point Likert scale for medical adequacy, conciseness, coherence, and comprehensibility. Moreover, consultants were asked whether the answers pose a risk to the patient’s safety. A modified Turing test was performed to distinguish responses generated by LLMs from those of doctors. Finally, the potential influence of the information generated by the LLMs on the raters’ own diagnosis and treatment opinions was evaluated.

Results

Over all categories, ORL doctors achieved superior ( P

Subjects

MeSH terms (MEDLINE)

OtolaryngologyAmbulatory CareProgramming LanguagesRetrospective StudiesHumansFemaleMaleOutpatientsAdultMiddle AgedLarge Language Models

Author keywords

artificial intelligencechatbotdigital healthglobal healthlarge language modelslow- and middle-income countriesotorhinolaryngologytelehealthtelemedicine

Authors and institutions

  • Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Langenbeckstraße 1, Mainz, 55131, Germany, +49 6131 17 7362.
  • School of Medicine, University of St Andrews, St Andrews, United Kingdom.
  • School of Computer Science, University of St Andrews, St Andrews, United Kingdom.
  • Institute for Digital Medicine, Philipps University Marburg, University Hospital Giessen and Marburg, Marburg, Germany.

ORCID profiles: Christoph Raphael Buhr, Christopher Seifen, Katharina Bahr-Hamm, Tilman Huppertz, Johannes Pordzik, Harry Smith, Tom Kelsey, Andrew Blaikie, Christoph Matthias, Sebastian Kuhn, Jonas Eckrich

Cite

APA

Buhr, C. R., Seifen, C., Bahr-Hamm, K., Huppertz, T., Pordzik, J., Smith, H., Kelsey, T., Blaikie, A., Matthias, C., Kuhn, S., & Eckrich, J. (2025). Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study. JMIR Formative Research. https://doi.org/10.2196/76896

Vancouver

Buhr CR, Seifen C, Bahr-Hamm K, Huppertz T, Pordzik J, Smith H, et al. Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study. JMIR Formative Research. 2025. doi:10.2196/76896

BibTeX
@article{Buhr2025,
  author = {Buhr, Christoph Raphael and Seifen, Christopher and Bahr-Hamm, Katharina and Huppertz, Tilman and Pordzik, Johannes and Smith, Harry and Kelsey, Tom and Blaikie, Andrew and Matthias, Christoph and Kuhn, Sebastian and Eckrich, Jonas},
  title = {Evaluating Locally Run Large Language Models (Gemma 2, Mistral Nemo, and Llama 3) for Outpatient Otorhinolaryngology Care: Retrospective Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/76896}
}
Start for free →