EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Quality of Machine Translations in Medical Texts: An Analysis Based on Standardised Evaluation Metrics

Pascal Block, Johanna Schaefer, Felix Maurer, Holger Storf

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti251380 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Pascal Block und 3 weiteren Autor:innen. Beteiligte Einrichtung: Goethe University Frankfurt, University Medicine, Institute of Medical Informatics (IMI), Frankfurt am Main, Germany. Hauptthemen laut MEDLINE: Translating, Natural Language Processing. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

The medical care of patients with rare diseases is a cross-border concern across the EU. This is also reflected in the usage statistics of the SE-ATLAS, where most access occurs via browser languages set to German, English, French, or Polish. The SE-ATLAS website provides information on healthcare services and patient organisations for rare diseases in Germany. As SE-ATLAS currently offers its content almost exclusively in German, non-German-speaking users may encounter language barriers. Against this background, this paper explores whether common machine translation systems can translate medical texts into other languages at a reasonable level of quality.

Methods

For this purpose, the translation systems DeepL, ChatGPT, and Google Translate were analysed. Translation quality was assessed using the standardised metrics BLEU, METEOR, and COMET. In contrast to subjective human assessments, these automated metrics allow for objective and reproducible evaluation. The analysis focused on machine-generated translations of German-language texts from the OPUS corpus into English, French, and Polish, each compared against existing reference translations.

Results

BLEU scores were generally lower than those of the other metrics, whereas METEOR and COMET indicated moderate to high translation quality. Translations into English were consistently rated higher than those into French and Polish.

Conclusion

As the three analysed translation systems showed hardly any statistically significant differences in translation quality and all delivered acceptable results, further criteria should be taken into account when choosing an appropriate system. These include factors such as data protection, cost-efficiency, and ease of integration.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

TranslatingNatural Language ProcessingHumansGermanyLanguage

Stichwörter der Autor:innen

MultilingualismRare DiseasesTranslations

Autor:innen und Institutionen

  • Goethe University Frankfurt, University Medicine, Institute of Medical Informatics (IMI), Frankfurt am Main, Germany.

Zitieren

APA

Block, P., Schaefer, J., Maurer, F., & Storf, H. (2025). Quality of Machine Translations in Medical Texts: An Analysis Based on Standardised Evaluation Metrics. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251380

Vancouver

Block P, Schaefer J, Maurer F, Storf H. Quality of Machine Translations in Medical Texts: An Analysis Based on Standardised Evaluation Metrics. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251380

BibTeX
@article{Block2025,
  author = {Block, Pascal and Schaefer, Johanna and Maurer, Felix and Storf, Holger},
  title = {Quality of Machine Translations in Medical Texts: An Analysis Based on Standardised Evaluation Metrics},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti251380}
}
Kostenlos starten →