EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French

Amandine Quercia, Jamil Zaghir, Christian Lovis, Christophe Gaudet-Blavignac

Studies in Health Technology and Informatics · 2024

DOI: 10.3233/shti240486 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2024. Verfasst von Amandine Quercia und 3 weiteren Autor:innen. Beteiligte Einrichtung: Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland. Hauptthemen laut MEDLINE: Benchmarking, Computer Simulation. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Generative Large Language Models (LLMs) have become ubiquitous in various fields, including healthcare and medicine. Consequently, there is growing interest in leveraging LLMs for medical applications, leading to the emergence of novel models daily. However, evaluation and benchmarking frameworks for LLMs are scarce, particularly those tailored for medical French. To address this gap, we introduce a minimal benchmark consisting of 114 open questions designed to assess the medical capabilities of LLMs in French. The proposed benchmark encompasses a wide range of medical domains, reflecting real-world clinical scenarios’ complexity. A preliminary validation involved testing seven widely used LLMs with a parameter size of 7 billion. Results revealed significant variability in performance, emphasizing the importance of rigorous evaluation before deploying LLMs in medical settings. In conclusion, we present a novel and valuable resource for rapidly evaluating LLMs in medical French. By promoting greater accountability and standardization, this benchmark has the potential to enhance trustworthiness and utility in harnessing LLMs for medical applications.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

BenchmarkingComputer SimulationFrance

Stichwörter der Autor:innen

Generative AILLMNLPbenchmark

Autor:innen und Institutionen

  • Division of Medical Information Sciences, Geneva University Hospitals, Geneva, Switzerland.
  • Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland.

Zitieren

APA

Quercia, A., Zaghir, J., Lovis, C., & Gaudet-Blavignac, C. (2024). MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti240486

Vancouver

Quercia A, Zaghir J, Lovis C, Gaudet-Blavignac C. MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French. Studies in Health Technology and Informatics. 2024. doi:10.3233/shti240486

BibTeX
@article{Quercia2024,
  author = {Quercia, Amandine and Zaghir, Jamil and Lovis, Christian and Gaudet-Blavignac, Christophe},
  title = {MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French},
  journal = {Studies in Health Technology and Informatics},
  year = {2024},
  doi = {10.3233/shti240486}
}
Kostenlos starten →