EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Performance Evaluation of Popular Open-Source Large Language Models in Healthcare

Saif Khairat, Tianyi Niu, John Geracitano, Zhaoqiang Zhou

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti250705 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Saif Khairat und 3 weiteren Autor:innen. Beteiligte Einrichtung: University of North Carolina at Chapel Hill. Hauptthemen laut MEDLINE: Programming Languages. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NCATS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper evaluated user preferences and performance metrics for two widely used open-source large language models (LLMs), Llama 3.1 8B and Mistral 3 Small 24B (AWQ), compared to the proprietary model GPT-4o, in the context of serving as a user-oriented healthcare assistant. The study highlighted the advantages of open-source LLMs, including transparency, cost-effectiveness, and customization potential for specific applications. A dual approach was used: first, ten participants ranked model-generated responses to various healthcare questions; second, computational performance metrics like response time, throughput, and time-to-first-token were benchmarked under different user loads. Results indicated that the majority of participants preferred GPT-4o responses; however, both open-source LLMs had relatively similar ratings. Furthermore, the benchmarking results underscored the efficiency and reliability of the models under load, showcasing their capabilities for real-world applications. This research contributed to understanding how open-source LLMs could meet the needs of diverse users across different domains, encouraging further exploration and adoption in various industries.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Programming LanguagesHumansBenchmarkingLarge Language Models

Stichwörter der Autor:innen

Large language modelsevaluationopen-source

Autor:innen und Institutionen

  • University of North Carolina at Chapel Hill.

Förderung

  • NCATS NIH HHS

Zitieren

APA

Khairat, S., Niu, T., Geracitano, J., & Zhou, Z. (2025). Performance Evaluation of Popular Open-Source Large Language Models in Healthcare. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250705

Vancouver

Khairat S, Niu T, Geracitano J, Zhou Z. Performance Evaluation of Popular Open-Source Large Language Models in Healthcare. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250705

BibTeX
@article{Khairat2025,
  author = {Khairat, Saif and Niu, Tianyi and Geracitano, John and Zhou, Zhaoqiang},
  title = {Performance Evaluation of Popular Open-Source Large Language Models in Healthcare},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti250705}
}
Kostenlos starten →