EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study

Yuriy Vasilev iD, Irina Raznitsyna iD, Anastasia Pamova iD, Tikhon Burtsev, Tatiana Bobrovskaya iD, Pavel Kosov iD, Anton Vladzymyrskyy iD, Olga Omelyanskaya iD, Kirill Arzamasov iD

Diagnostics · 19.12.2025 · 16 (1) · 3

DOI: 10.3390/diagnostics16010003 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Diagnostics, online erschienen am 19.12.2025. Verfasst von Yuriy Vasilev und 8 weiteren Autor:innen. Beteiligte Einrichtung: Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: This paper was prepared by a team of authors within the framework of a scientific and practical project in the field of medicine "A promising automated workpla…, A promising automated workplace of a radiologist based on generative artificial intelligence.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Electronic health records (EHRs) remain a vital source of clinical information, yet processing these heterogeneous data is extremely labor-intensive. Summarization of these data using Large Language Models (LLMs) is considered a promising tool to support practicing physicians. Unbiased, automated quality control is crucial for integrating the tools into routine practice, saving time and labor. This pilot study aimed to assess the potential and constraints of self-contained evaluation of summarization quality (without expert involvement) based on automatic evaluation metrics and LLM-as-a-judge.

Methods

The summaries of text data from 30 EHRs were generated by six open-source low-parameter LLMs. The medical summaries were evaluated using standard automatic metrics (BLEU, ROUGE, METEOR, BERTScore) as well as the LLM-as-a-judge approach using the following criteria: relevance, completeness, redundancy, coherence and structure, grammar and terminology, and hallucinations. Expert evaluation was conducted using the same criteria.

Results

The results showed that LLMs hold great promise for summarizing medical data. Nevertheless, neither the evaluation metrics nor LLM judges are reliable in detecting factual errors and semantic distortions (hallucinations). In terms of relevance, the Pearson correlation between the summary quality score and the expert opinions was 0.688.

Conclusions

Completely automating the evaluation of medical summaries remains challenging. Further research should focus on dedicated methods for detecting hallucinations, along with investigating larger or specialized models trained on medical texts. Additionally, the potential integration of retrieval-augmented generation (RAG) within the LLM-as-a-judge architecture deserves attention. Nevertheless, even now, the combination of LLMs and the automatic evaluation metrics can underpin medical decision support systems by performing initial evaluations and highlighting potential shortcomings for expert review.

Schlagwörter

Stichwörter der Autor:innen

LLM-as-a-judgeelectronic health recordslarge language modelsummaries

Autor:innen und Institutionen

  • Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia.
  • Institute of Artificial Intelligence, MIREA-Russian Technological University, 119454 Moscow, Russia.
  • Department of Information Technology and Medical Data Processing, Federal State I.M. Sechenov First Moscow State Medical University of the Ministry of Health of the Russian Federation (Sechenov University), 119991 Mosco…

ORCID-Profile: Yuriy Vasilev, Irina Raznitsyna, Anastasia Pamova, Tatiana Bobrovskaya, Pavel Kosov, Anton Vladzymyrskyy, Olga Omelyanskaya, Kirill Arzamasov

Förderung

  • This paper was prepared by a team of authors within the framework of a scientific and practical project in the field of medicine "A promising automated workpla…
  • A promising automated workplace of a radiologist based on generative artificial intelligence

Zitieren

APA

Vasilev, Y., Raznitsyna, I., Pamova, A., Burtsev, T., Bobrovskaya, T., Kosov, P., Vladzymyrskyy, A., Omelyanskaya, O., & Arzamasov, K. (2025). Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study. Diagnostics, 16(1), 3. https://doi.org/10.3390/diagnostics16010003

Vancouver

Vasilev Y, Raznitsyna I, Pamova A, Burtsev T, Bobrovskaya T, Kosov P, et al. Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study. Diagnostics. 2025;16(1):3. doi:10.3390/diagnostics16010003

BibTeX
@article{Vasilev2025,
  author = {Vasilev, Yuriy and Raznitsyna, Irina and Pamova, Anastasia and Burtsev, Tikhon and Bobrovskaya, Tatiana and Kosov, Pavel and Vladzymyrskyy, Anton and Omelyanskaya, Olga and Arzamasov, Kirill},
  title = {Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study},
  journal = {Diagnostics},
  year = {2025},
  volume = {16},
  number = {1},
  pages = {3},
  doi = {10.3390/diagnostics16010003}
}
Kostenlos starten →