EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Evaluating Medium Scale, Open-Source Large Language Models: Towards Decision Support in a Precision Oncology Care Delivery Context

Kevin Kaufmes, Georg Mathes, Dilyana Vladimirova, Stephanie Berger, Christian Fegeler, Stefan Sigle

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti251382 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Kevin Kaufmes und 5 weiteren Autor:innen. Beteiligte Einrichtung: MOLIT Institute, Heilbronn, Germany. Hauptthemen laut MEDLINE: Precision Medicine, Medical Oncology, Decision Support Systems, Clinical, Programming Languages. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

In the context of precision oncology, patients often have complex conditions that require treatment based on specific and up-to-date knowledge of guidelines and research. This entails considerable effort when preparing such cases for molecular tumor boards (MTBs). Large language models (LLMs) could help to lower this burden if they could provide such information quickly and precisely on demand. Since out-of-the-box LLMs are not specialized for clinical contexts, this work aims to investigate their usefulness for answering questions arising during MTB preparation. As such questions can contain sensitive data, we evaluated medium-scale models suitable for running on-premise using consumer grade hardware.

Methods

Three recent LLMs to be tested were selected based on established benchmarks and unique characteristics like reasoning capability. Exemplary questions related to MTBs were collected from domain experts. Six of those were selected for the LLMs to generate responses to. Response quality and correctness was evaluated by experts using a questionnaire.

Results

Out of 60 contacted domain experts, 5 fully completed the survey, with another 5 completing it partially. The evaluation revealed a modest overall performance. Our findings identified significant issues, where a large percentage of answers contained outdated or incomplete information, as well as factual errors. Additionally, a high discordance between evaluators regarding correctness and varying rater confidence has been observed.

Conclusion

Our results seem to be indicating that medium-scale LLMs are currently insufficiently reliable for use in precision oncology. Common issues include outdated information and confident presentation of misinformation, which indicates a gap between benchmark- and real-world performance. Future research should focus on mitigating limitations with advanced techniques such as Retrieval-Augmented-Generation (RAG), web search capability or advanced prompting, while prioritizing patient safety.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Precision MedicineMedical OncologyDecision Support Systems, ClinicalProgramming LanguagesNeoplasmsHumansLarge Language Models

Stichwörter der Autor:innen

BenchmarkingClinical Decision Support SystemsLarge Language ModelsMedical OncologyPrecision Medicine

Autor:innen und Institutionen

  • MOLIT Institute, Heilbronn, Germany.
  • SLK Clinics, Heilbronn, Germany.
  • University of Heilbronn, Heilbronn, Germany.

Zitieren

APA

Kaufmes, K., Mathes, G., Vladimirova, D., Berger, S., Fegeler, C., & Sigle, S. (2025). Evaluating Medium Scale, Open-Source Large Language Models: Towards Decision Support in a Precision Oncology Care Delivery Context. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251382

Vancouver

Kaufmes K, Mathes G, Vladimirova D, Berger S, Fegeler C, Sigle S. Evaluating Medium Scale, Open-Source Large Language Models: Towards Decision Support in a Precision Oncology Care Delivery Context. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251382

BibTeX
@article{Kaufmes2025,
  author = {Kaufmes, Kevin and Mathes, Georg and Vladimirova, Dilyana and Berger, Stephanie and Fegeler, Christian and Sigle, Stefan},
  title = {Evaluating Medium Scale, Open-Source Large Language Models: Towards Decision Support in a Precision Oncology Care Delivery Context},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti251382}
}
Kostenlos starten →