Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation
JMIR AI · 04.12.2025 · 4 · e72153-e72153
DOI: 10.2196/72153 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR AI, online erschienen am 04.12.2025. Verfasst von Veysel Kocaman und 3 weiteren Autor:innen. Beteiligte Einrichtung: John Snow Labs Inc, 16192 Coastal Highway, Lewes, DE, 19958, United States, +1 (302) 786-5227. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The proliferation of both general purpose and health care–specific large language models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks, self-preference distorts LLM-as-a-judge approaches, and there is a gap between the tasks used to test models and those used in clinical practice.
Objective
In response, we propose CLEVER (Clinical Large Language Model Evaluation–Expert Review), a methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks.
Methods
We demonstrate the methodology by comparing GPT-4o (OpenAI) against 2 health care–specific LLMs, with 8 billion and 70 billion parameters, over 3 tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research.
Results
Medical doctors prefer the medical model–small LLM trained by John Snow Labs over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness.
Conclusions
The models show comparable performance on open-ended medical question answering, suggesting that health care–specific LLMs can outperform much larger general purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting interannotator agreement, interclass correlation, and washout period analysis.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- John Snow Labs Inc, 16192 Coastal Highway, Lewes, DE, 19958, United States, +1 (302) 786-5227.
- Computational Sciences and Informatics (CSI), George Mason University, Fairfax, VA, United States.
Zitieren
APA
Kocaman, V., Kaya, M. A., Feier, A. M., & Talby, D. (2025). Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI, 4, e72153-e72153. https://doi.org/10.2196/72153
Vancouver
Kocaman V, Kaya MA, Feier AM, Talby D. Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI. 2025;4:e72153-e72153. doi:10.2196/72153
BibTeX
@article{Kocaman2025,
author = {Kocaman, Veysel and Kaya, Mustafa Aytuğ and Feier, Andrei Marian and Talby, David},
title = {Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation},
journal = {JMIR AI},
year = {2025},
volume = {4},
pages = {e72153--e72153},
doi = {10.2196/72153}
}
EUVIMED