EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Comparative Evaluation of Large Language Models for Translating Radiology Reports into Hindi

Amit Gupta, Ashish Rastogi, Hema Malhotra, Krithika Rangarajan

Indian Journal of Radiology and Imaging · 01.01.2025 · 35 (01) · 088-096

DOI: 10.1055/s-0044-1789618 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Indian Journal of Radiology and Imaging, erschienen am 01.01.2025. Verfasst von Amit Gupta und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Dr. Bhim Rao Ambedkar Institute-Rotary Cancer Hospital, All India Institute of Medical Sciences, New Delhi. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

The aim of this study was to compare the performance of four publicly available large language models (LLMs)—GPT-4o, GPT-4, Gemini, and Claude Opus—in translating radiology reports into simple Hindi. Materials and Methods In this retrospective study, 100 computed tomography (CT) scan report impressions were gathered from a tertiary care cancer center. Reference translations of these impressions into simple Hindi were done by a bilingual radiology staff in consultation with a radiologist. Two distinct prompts were used to assess the LLMs' ability to translate these report impressions into simple Hindi. Translated reports were assessed by a radiologist for instances of misinterpretation, omission, and addition of fictitious information. Translation quality was assessed using Bilingual Evaluation Understudy (BLEU), Metric for Evaluation of Translation with Explicit ORdering (METEOR), Translation Edit Rate (TER), and character F-score (CHRF) scores. Statistical analyses were performed to compare the LLM performance across prompts.

Results

Nine instances of misinterpretation and two instances of omission of information were found on radiologist evaluation of the total 800 LLM-generated translated report impressions. For prompt 1, Gemini outperformed others in BLEU (p < 0.001) and METEOR scores (p = 0.001), and was superior to GPT-4o and GPT-4 in TER and CHRF (p < 0.001), but comparable to Claude (p = 0.501 for TER and p = 0.90 for CHRF). For prompt 2, GPT-4o outperformed all others (p < 0.001) in all metrics. Prompt 2 yielded better BLEU, METEOR, and CHRF scores (p < 0.001), while prompt 1 had a better TER score (p < 0.001).

Conclusion

While each LLM's effectiveness varied with prompt wording, all models demonstrated potential in translating and simplifying radiology report impressions.

Schlagwörter

Stichwörter der Autor:innen

BLEU scoreChatGPTlarge language modelsradiology reports

Autor:innen und Institutionen

  • Department of Radiology, Dr. Bhim Rao Ambedkar Institute-Rotary Cancer Hospital, All India Institute of Medical Sciences, New Delhi.

Zitieren

APA

Gupta, A., Rastogi, A., Malhotra, H., & Rangarajan, K. (2025). Comparative Evaluation of Large Language Models for Translating Radiology Reports into Hindi. Indian Journal of Radiology and Imaging, 35(01), 088-096. https://doi.org/10.1055/s-0044-1789618

Vancouver

Gupta A, Rastogi A, Malhotra H, Rangarajan K. Comparative Evaluation of Large Language Models for Translating Radiology Reports into Hindi. Indian Journal of Radiology and Imaging. 2025;35(01):088-096. doi:10.1055/s-0044-1789618

BibTeX
@article{Gupta2025,
  author = {Gupta, Amit and Rastogi, Ashish and Malhotra, Hema and Rangarajan, Krithika},
  title = {Comparative Evaluation of Large Language Models for Translating Radiology Reports into Hindi},
  journal = {Indian Journal of Radiology and Imaging},
  year = {2025},
  volume = {35},
  number = {01},
  pages = {088--096},
  doi = {10.1055/s-0044-1789618}
}
Kostenlos starten →