Benchmarking Large Language Models for Italian Medical Text Classification: Are Generative Models the Best Choice?
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti251486 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Livia Lilli und 4 weiteren Autor:innen. Beteiligte Einrichtung: Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy. Hauptthemen laut MEDLINE: Natural Language Processing, Benchmarking, Breast Neoplasms, Electronic Health Records. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
The extraction of meaningful information from clinical reports has been an area of growing interest, with a variety of studies leveraging natural language processing (NLP) techniques based on BERT architectures and generative large language models (LLMs). However, identifying the most effective approach remains challenging, especially for text classification, where model architecture, data availability, domain-specific nuances and language play a crucial role in performance. In this study, we present a benchmark analysis of generative LLMs and BERT-based models for the classification of metastasis in Italian clinical reports of breast cancer patients. Our methodology compares the performance of generative LLMs implemented within a structured generation framework, versus BERT-based models fine-tuned on the metastasis classification task, and also applied in a zero-shot learning setting. In our experiments, fine-tuned BERT models achieved the most balanced results (F1 = 0.884, AUC = 0.720). Generative LLMs showed promising performance, with potential for improvement through further adaptation. Finally, our study suggests that both BERT-based models and generative LLMs are potential solutions also in low computational settings, making them accessible for real-world clinical applications, particularly in medical text classification.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy.
- Catholic University of the Sacred Heart, Rome, Italy.
Zitieren
APA
Lilli, L., Patarnello, S., Masciocchi, C., Marchetti, A., & Arcuri, G. (2025). Benchmarking Large Language Models for Italian Medical Text Classification: Are Generative Models the Best Choice?. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251486
Vancouver
Lilli L, Patarnello S, Masciocchi C, Marchetti A, Arcuri G. Benchmarking Large Language Models for Italian Medical Text Classification: Are Generative Models the Best Choice?. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251486
BibTeX
@article{Lilli2025,
author = {Lilli, Livia and Patarnello, Stefano and Masciocchi, Carlotta and Marchetti, Antonio and Arcuri, Giovanni},
title = {Benchmarking Large Language Models for Italian Medical Text Classification: Are Generative Models the Best Choice?},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti251486}
}
EUVIMED