MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French
Studies in Health Technology and Informatics · 2024
DOI: 10.3233/shti240486 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2024. Verfasst von Amandine Quercia und 3 weiteren Autor:innen. Beteiligte Einrichtung: Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland. Hauptthemen laut MEDLINE: Benchmarking, Computer Simulation. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Generative Large Language Models (LLMs) have become ubiquitous in various fields, including healthcare and medicine. Consequently, there is growing interest in leveraging LLMs for medical applications, leading to the emergence of novel models daily. However, evaluation and benchmarking frameworks for LLMs are scarce, particularly those tailored for medical French. To address this gap, we introduce a minimal benchmark consisting of 114 open questions designed to assess the medical capabilities of LLMs in French. The proposed benchmark encompasses a wide range of medical domains, reflecting real-world clinical scenarios’ complexity. A preliminary validation involved testing seven widely used LLMs with a parameter size of 7 billion. Results revealed significant variability in performance, emphasizing the importance of rigorous evaluation before deploying LLMs in medical settings. In conclusion, we present a novel and valuable resource for rapidly evaluating LLMs in medical French. By promoting greater accountability and standardization, this benchmark has the potential to enhance trustworthiness and utility in harnessing LLMs for medical applications.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland.
- Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland.
Zitieren
APA
Quercia, A., Zaghir, J., Lovis, C., & Gaudet-Blavignac, C. (2024). MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti240486
Vancouver
Quercia A, Zaghir J, Lovis C, Gaudet-Blavignac C. MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French. Studies in Health Technology and Informatics. 2024. doi:10.3233/shti240486
BibTeX
@article{Quercia2024,
author = {Quercia, Amandine and Zaghir, Jamil and Lovis, Christian and Gaudet-Blavignac, Christophe},
title = {MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French},
journal = {Studies in Health Technology and Informatics},
year = {2024},
doi = {10.3233/shti240486}
}
EUVIMED