Quality of Machine Translations in Medical Texts: An Analysis Based on Standardised Evaluation Metrics
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti251380 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Pascal Block und 3 weiteren Autor:innen. Beteiligte Einrichtung: Goethe University Frankfurt, University Medicine, Institute of Medical Informatics (IMI), Frankfurt am Main, Germany. Hauptthemen laut MEDLINE: Translating, Natural Language Processing. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Introduction
The medical care of patients with rare diseases is a cross-border concern across the EU. This is also reflected in the usage statistics of the SE-ATLAS, where most access occurs via browser languages set to German, English, French, or Polish. The SE-ATLAS website provides information on healthcare services and patient organisations for rare diseases in Germany. As SE-ATLAS currently offers its content almost exclusively in German, non-German-speaking users may encounter language barriers. Against this background, this paper explores whether common machine translation systems can translate medical texts into other languages at a reasonable level of quality.
Methods
For this purpose, the translation systems DeepL, ChatGPT, and Google Translate were analysed. Translation quality was assessed using the standardised metrics BLEU, METEOR, and COMET. In contrast to subjective human assessments, these automated metrics allow for objective and reproducible evaluation. The analysis focused on machine-generated translations of German-language texts from the OPUS corpus into English, French, and Polish, each compared against existing reference translations.
Results
BLEU scores were generally lower than those of the other metrics, whereas METEOR and COMET indicated moderate to high translation quality. Translations into English were consistently rated higher than those into French and Polish.
Conclusion
As the three analysed translation systems showed hardly any statistically significant differences in translation quality and all delivered acceptable results, further criteria should be taken into account when choosing an appropriate system. These include factors such as data protection, cost-efficiency, and ease of integration.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Goethe University Frankfurt, University Medicine, Institute of Medical Informatics (IMI), Frankfurt am Main, Germany.
Zitieren
APA
Block, P., Schaefer, J., Maurer, F., & Storf, H. (2025). Quality of Machine Translations in Medical Texts: An Analysis Based on Standardised Evaluation Metrics. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251380
Vancouver
Block P, Schaefer J, Maurer F, Storf H. Quality of Machine Translations in Medical Texts: An Analysis Based on Standardised Evaluation Metrics. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251380
BibTeX
@article{Block2025,
author = {Block, Pascal and Schaefer, Johanna and Maurer, Felix and Storf, Holger},
title = {Quality of Machine Translations in Medical Texts: An Analysis Based on Standardised Evaluation Metrics},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti251380}
}
EUVIMED