Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation
JMIR AI · 2025-12-04 · 4 · e72153-e72153
DOI: 10.2196/72153 ↗At a glance
Journal article in JMIR AI, published online on 2025-12-04. Written by Veysel Kocaman and 3 co-authors. Institution: John Snow Labs Inc, 16192 Coastal Highway, Lewes, DE, 19958, United States, +1 (302) 786-5227. Cited 35 times so far (Crossref); 34 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
The proliferation of both general purpose and health care–specific large language models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks, self-preference distorts LLM-as-a-judge approaches, and there is a gap between the tasks used to test models and those used in clinical practice.
Objective
In response, we propose CLEVER (Clinical Large Language Model Evaluation–Expert Review), a methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks.
Methods
We demonstrate the methodology by comparing GPT-4o (OpenAI) against 2 health care–specific LLMs, with 8 billion and 70 billion parameters, over 3 tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research.
Results
Medical doctors prefer the medical model–small LLM trained by John Snow Labs over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness.
Conclusions
The models show comparable performance on open-ended medical question answering, suggesting that health care–specific LLMs can outperform much larger general purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting interannotator agreement, interclass correlation, and washout period analysis.
Subjects
Author keywords
Authors and institutions
- John Snow Labs Inc, 16192 Coastal Highway, Lewes, DE, 19958, United States, +1 (302) 786-5227.
- Computational Sciences and Informatics (CSI), George Mason University, Fairfax, VA, United States.
Cite
APA
Kocaman, V., Kaya, M. A., Feier, A. M., & Talby, D. (2025). Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI, 4, e72153-e72153. https://doi.org/10.2196/72153
Vancouver
Kocaman V, Kaya MA, Feier AM, Talby D. Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI. 2025;4:e72153-e72153. doi:10.2196/72153
BibTeX
@article{Kocaman2025,
author = {Kocaman, Veysel and Kaya, Mustafa Aytuğ and Feier, Andrei Marian and Talby, David},
title = {Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation},
journal = {JMIR AI},
year = {2025},
volume = {4},
pages = {e72153--e72153},
doi = {10.2196/72153}
}
EUVIMED