Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study
Diagnostics · 19.12.2025 · 16 (1) · 3
DOI: 10.3390/diagnostics16010003 ↗Kurz gefasst
Fachartikel in der Zeitschrift Diagnostics, online erschienen am 19.12.2025. Verfasst von Yuriy Vasilev und 8 weiteren Autor:innen. Beteiligte Einrichtung: Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: This paper was prepared by a team of authors within the framework of a scientific and practical project in the field of medicine "A promising automated workpla…, A promising automated workplace of a radiologist based on generative artificial intelligence.
Abstract (Originalsprache)
Background
Electronic health records (EHRs) remain a vital source of clinical information, yet processing these heterogeneous data is extremely labor-intensive. Summarization of these data using Large Language Models (LLMs) is considered a promising tool to support practicing physicians. Unbiased, automated quality control is crucial for integrating the tools into routine practice, saving time and labor. This pilot study aimed to assess the potential and constraints of self-contained evaluation of summarization quality (without expert involvement) based on automatic evaluation metrics and LLM-as-a-judge.
Methods
The summaries of text data from 30 EHRs were generated by six open-source low-parameter LLMs. The medical summaries were evaluated using standard automatic metrics (BLEU, ROUGE, METEOR, BERTScore) as well as the LLM-as-a-judge approach using the following criteria: relevance, completeness, redundancy, coherence and structure, grammar and terminology, and hallucinations. Expert evaluation was conducted using the same criteria.
Results
The results showed that LLMs hold great promise for summarizing medical data. Nevertheless, neither the evaluation metrics nor LLM judges are reliable in detecting factual errors and semantic distortions (hallucinations). In terms of relevance, the Pearson correlation between the summary quality score and the expert opinions was 0.688.
Conclusions
Completely automating the evaluation of medical summaries remains challenging. Further research should focus on dedicated methods for detecting hallucinations, along with investigating larger or specialized models trained on medical texts. Additionally, the potential integration of retrieval-augmented generation (RAG) within the LLM-as-a-judge architecture deserves attention. Nevertheless, even now, the combination of LLMs and the automatic evaluation metrics can underpin medical decision support systems by performing initial evaluations and highlighting potential shortcomings for expert review.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia.
- Institute of Artificial Intelligence, MIREA-Russian Technological University, 119454 Moscow, Russia.
- Department of Information Technology and Medical Data Processing, Federal State I.M. Sechenov First Moscow State Medical University of the Ministry of Health of the Russian Federation (Sechenov University), 119991 Mosco…
ORCID-Profile: Yuriy Vasilev, Irina Raznitsyna, Anastasia Pamova, Tatiana Bobrovskaya, Pavel Kosov, Anton Vladzymyrskyy, Olga Omelyanskaya, Kirill Arzamasov
Förderung
- This paper was prepared by a team of authors within the framework of a scientific and practical project in the field of medicine "A promising automated workpla…
- A promising automated workplace of a radiologist based on generative artificial intelligence
Zitieren
APA
Vasilev, Y., Raznitsyna, I., Pamova, A., Burtsev, T., Bobrovskaya, T., Kosov, P., Vladzymyrskyy, A., Omelyanskaya, O., & Arzamasov, K. (2025). Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study. Diagnostics, 16(1), 3. https://doi.org/10.3390/diagnostics16010003
Vancouver
Vasilev Y, Raznitsyna I, Pamova A, Burtsev T, Bobrovskaya T, Kosov P, et al. Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study. Diagnostics. 2025;16(1):3. doi:10.3390/diagnostics16010003
BibTeX
@article{Vasilev2025,
author = {Vasilev, Yuriy and Raznitsyna, Irina and Pamova, Anastasia and Burtsev, Tikhon and Bobrovskaya, Tatiana and Kosov, Pavel and Vladzymyrskyy, Anton and Omelyanskaya, Olga and Arzamasov, Kirill},
title = {Evaluating Medical Text Summaries Using Automatic Evaluation Metrics and LLM-as-a-Judge Approach: A Pilot Study},
journal = {Diagnostics},
year = {2025},
volume = {16},
number = {1},
pages = {3},
doi = {10.3390/diagnostics16010003}
}
EUVIMED