EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Benchmarking Large Language Models for Italian Medical Text Classification: Are Generative Models the Best Choice?

Livia Lilli, Stefano Patarnello, Carlotta Masciocchi, Antonio Marchetti, Giovanni Arcuri

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti251486 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Livia Lilli und 4 weiteren Autor:innen. Beteiligte Einrichtung: Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy. Hauptthemen laut MEDLINE: Natural Language Processing, Benchmarking, Breast Neoplasms, Electronic Health Records. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The extraction of meaningful information from clinical reports has been an area of growing interest, with a variety of studies leveraging natural language processing (NLP) techniques based on BERT architectures and generative large language models (LLMs). However, identifying the most effective approach remains challenging, especially for text classification, where model architecture, data availability, domain-specific nuances and language play a crucial role in performance. In this study, we present a benchmark analysis of generative LLMs and BERT-based models for the classification of metastasis in Italian clinical reports of breast cancer patients. Our methodology compares the performance of generative LLMs implemented within a structured generation framework, versus BERT-based models fine-tuned on the metastasis classification task, and also applied in a zero-shot learning setting. In our experiments, fine-tuned BERT models achieved the most balanced results (F1 = 0.884, AUC = 0.720). Generative LLMs showed promising performance, with potential for improvement through further adaptation. Finally, our study suggests that both BERT-based models and generative LLMs are potential solutions also in low computational settings, making them accessible for real-world clinical applications, particularly in medical text classification.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingBenchmarkingBreast NeoplasmsElectronic Health RecordsData MiningItalyHumansFemaleLarge Language Models

Stichwörter der Autor:innen

Clinical ReportLarge Language ModelsNatural Language Processing

Autor:innen und Institutionen

  • Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy.
  • Catholic University of the Sacred Heart, Rome, Italy.

Zitieren

APA

Lilli, L., Patarnello, S., Masciocchi, C., Marchetti, A., & Arcuri, G. (2025). Benchmarking Large Language Models for Italian Medical Text Classification: Are Generative Models the Best Choice?. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251486

Vancouver

Lilli L, Patarnello S, Masciocchi C, Marchetti A, Arcuri G. Benchmarking Large Language Models for Italian Medical Text Classification: Are Generative Models the Best Choice?. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251486

BibTeX
@article{Lilli2025,
  author = {Lilli, Livia and Patarnello, Stefano and Masciocchi, Carlotta and Marchetti, Antonio and Arcuri, Giovanni},
  title = {Benchmarking Large Language Models for Italian Medical Text Classification: Are Generative Models the Best Choice?},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti251486}
}
Kostenlos starten →