Performance Evaluation of Popular Open-Source Large Language Models in Healthcare
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti250705 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Saif Khairat und 3 weiteren Autor:innen. Beteiligte Einrichtung: University of North Carolina at Chapel Hill. Hauptthemen laut MEDLINE: Programming Languages. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NCATS NIH HHS.
Abstract (Originalsprache)
This paper evaluated user preferences and performance metrics for two widely used open-source large language models (LLMs), Llama 3.1 8B and Mistral 3 Small 24B (AWQ), compared to the proprietary model GPT-4o, in the context of serving as a user-oriented healthcare assistant. The study highlighted the advantages of open-source LLMs, including transparency, cost-effectiveness, and customization potential for specific applications. A dual approach was used: first, ten participants ranked model-generated responses to various healthcare questions; second, computational performance metrics like response time, throughput, and time-to-first-token were benchmarked under different user loads. Results indicated that the majority of participants preferred GPT-4o responses; however, both open-source LLMs had relatively similar ratings. Furthermore, the benchmarking results underscored the efficiency and reliability of the models under load, showcasing their capabilities for real-world applications. This research contributed to understanding how open-source LLMs could meet the needs of diverse users across different domains, encouraging further exploration and adoption in various industries.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- University of North Carolina at Chapel Hill.
Förderung
- NCATS NIH HHS
Zitieren
APA
Khairat, S., Niu, T., Geracitano, J., & Zhou, Z. (2025). Performance Evaluation of Popular Open-Source Large Language Models in Healthcare. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250705
Vancouver
Khairat S, Niu T, Geracitano J, Zhou Z. Performance Evaluation of Popular Open-Source Large Language Models in Healthcare. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250705
BibTeX
@article{Khairat2025,
author = {Khairat, Saif and Niu, Tianyi and Geracitano, John and Zhou, Zhaoqiang},
title = {Performance Evaluation of Popular Open-Source Large Language Models in Healthcare},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti250705}
}
EUVIMED