EUVIMEDEUVIMED entdecken
Open Access

Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation

Veysel Kocaman, Mustafa Aytuğ Kaya, Andrei Marian Feier, David Talby

JMIR AI · 04.12.2025 · 4 · e72153-e72153

DOI: 10.2196/72153 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR AI, online erschienen am 04.12.2025. Verfasst von Veysel Kocaman und 3 weiteren Autor:innen. Beteiligte Einrichtung: John Snow Labs Inc, 16192 Coastal Highway, Lewes, DE, 19958, United States, +1 (302) 786-5227. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The proliferation of both general purpose and health care–specific large language models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks, self-preference distorts LLM-as-a-judge approaches, and there is a gap between the tasks used to test models and those used in clinical practice.

Objective

In response, we propose CLEVER (Clinical Large Language Model Evaluation–Expert Review), a methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks.

Methods

We demonstrate the methodology by comparing GPT-4o (OpenAI) against 2 health care–specific LLMs, with 8 billion and 70 billion parameters, over 3 tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research.

Results

Medical doctors prefer the medical model–small LLM trained by John Snow Labs over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness.

Conclusions

The models show comparable performance on open-ended medical question answering, suggesting that health care–specific LLMs can outperform much larger general purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting interannotator agreement, interclass correlation, and washout period analysis.

Schlagwörter

Stichwörter der Autor:innen

NLPartificial intelligenceclinical relevanceevaluation frameworkfactualitygenerative AIlarge language modelsmedical LLMsnatural language processing

Autor:innen und Institutionen

  • John Snow Labs Inc, 16192 Coastal Highway, Lewes, DE, 19958, United States, +1 (302) 786-5227.
  • Computational Sciences and Informatics (CSI), George Mason University, Fairfax, VA, United States.

Zitieren

APA

Kocaman, V., Kaya, M. A., Feier, A. M., & Talby, D. (2025). Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI, 4, e72153-e72153. https://doi.org/10.2196/72153

Vancouver

Kocaman V, Kaya MA, Feier AM, Talby D. Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI. 2025;4:e72153-e72153. doi:10.2196/72153

BibTeX
@article{Kocaman2025,
  author = {Kocaman, Veysel and Kaya, Mustafa Aytuğ and Feier, Andrei Marian and Talby, David},
  title = {Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation},
  journal = {JMIR AI},
  year = {2025},
  volume = {4},
  pages = {e72153--e72153},
  doi = {10.2196/72153}
}
Kostenlos starten →