EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluation of six generative large language models answering preanalytical questions in medical microbiology

Michelle Bressan, Eline Nadieh Meijer, Vladimira Hinic, Tilia Ellendorff, Jean-Philippe Goldman, Tannon Kew, Oliver Nolte, Gerold Schneider, Adrian Egli, LLM Evaluation Consortium

Clinical Microbiology and Infection · 2026-10

DOI: 10.1016/j.cmi.2026.09.035 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Clinical Microbiology and Infection, 2026-10. Verfasst von Michelle Bressan und 9 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Medical Microbiology, University of Zurich, Zurich, Switzerland. Die Arbeit nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: University of Zurich, Swiss National Science Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

OBJECTIVES: Large language models (LLMs) have the potential to be integrated into diagnostics workflows, yet their reliability in preanalytical diagnostic stewardship remains uncertain. We aimed to evaluate the performance of six LLMs answering common preanalytical questions in the domain of medical microbiology. Next, we proposed a blueprint to evaluate LLMs in medical microbiology. METHODS: We compared general-purpose models from OpenAI (generative pretrained transformer [GPT]-3.5, GPT-4) and Meta artificial intelligence (AI) (Llama2-7B, Llama2-13B), alongside two domain-enhanced versions (customized GPT-4 agent and Llama2-7B Medical) in a hybrid evaluation framework. First, we quantified the response length (word count), then assessed the similarity between LLM-generated responses and expert-reviewed reference answers (BLEU/ROUGE scores, BERTScore). Finally, 33 medical professionals rated the accuracy and completeness of the LLM-generated responses on a 1-7-point scale. RESULTS: Responses of OpenAI models showed greater similarity to the reference set and significantly outperformed Meta AI-based models in accuracy and completeness. The customized GPT matched performance of OpenAI's general-purpose models while generating significantly shorter responses (p

Schlagwörter

Stichwörter der Autor:innen

Artificial intelligenceBenchmarkChatGPTDiagnosticLarge language modelLlamaMedical microbiologyMetaOpenAIPreanalytic

Autor:innen und Institutionen

  • Institute of Medical Microbiology, University of Zurich, Zurich, Switzerland.
  • Linguistics Research Infrastructure, University of Zurich, Zurich, Switzerland.
  • Institute of Medical Microbiology, University of Zurich, Zurich, Switzerland. Electronic address: aegli@imm.uzh.ch.

Förderung

  • University of Zurich
  • Swiss National Science Foundation

Zitieren

APA

Bressan, M., Meijer, E. N., Hinic, V., Ellendorff, T., Goldman, J. P., Kew, T., Nolte, O., Schneider, G., Egli, A., & LLM Evaluation Consortium (2026). Evaluation of six generative large language models answering preanalytical questions in medical microbiology. Clinical Microbiology and Infection. https://doi.org/10.1016/j.cmi.2026.09.035

Vancouver

Bressan M, Meijer EN, Hinic V, Ellendorff T, Goldman JP, Kew T, et al. Evaluation of six generative large language models answering preanalytical questions in medical microbiology. Clinical Microbiology and Infection. 2026. doi:10.1016/j.cmi.2026.09.035

BibTeX
@article{Bressan2026,
  author = {Bressan, Michelle and Meijer, Eline Nadieh and Hinic, Vladimira and Ellendorff, Tilia and Goldman, Jean-Philippe and Kew, Tannon and Nolte, Oliver and Schneider, Gerold and Egli, Adrian and LLM Evaluation Consortium, LLM Evaluation Consortium},
  title = {Evaluation of six generative large language models answering preanalytical questions in medical microbiology},
  journal = {Clinical Microbiology and Infection},
  year = {2026},
  doi = {10.1016/j.cmi.2026.09.035}
}
Kostenlos starten →