Evaluating a Large Language Model in Translating Patient Instructions to Spanish Using a Standardized Framework
JAMA Pediatrics · 01.09.2025 · 179 (9) · 1026
DOI: 10.1001/jamapediatrics.2025.1729 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMA Pediatrics, erschienen am 01.09.2025. Verfasst von Mondira Ray und 12 weiteren Autor:innen. Beteiligte Einrichtung: Division of General Pediatrics, Boston Children's Hospital, Boston, Massachusetts. Hauptthemen laut MEDLINE: Translating, Language, Communication Barriers. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NLM NIH HHS.
Abstract (Originalsprache)
Importance Patients and caregivers who use languages other than English in the US encounter barriers to accessing language-concordant written instructions after clinical visits. Large language models (LLMs), such as OpenAI’s GPT-4o, may improve access to translated patient materials; however, rigorous evaluation is needed to ensure clinical standards are met.
Objective
To determine whether GPT-4o can generate high-quality Spanish translations of personalized patient instructions comparable to those performed by professional human translators. Design, Setting, and Participants This cross-sectional study compared LLM translations to professional human translations using equivalence testing. The personalized pediatric instructions used were derived from real clinical encounters at a large US academic medical center and translated between January 2023 and December 2023. Patient instructions in English were translated into Spanish by GPT-4o and professional human translators. The source English texts were translated using GPT-4o on August 2, 2024. Both sets of translations were evaluated by 3 independent professional medical translators. Exposure Patient instructions were translated using GPT-4o with an engineered prompt, and these translations were compared with those produced by professional human translators. Main Outcomes and Measures The primary outcome was translation quality, assessed using the Multidimensional Quality Metrics (MQM) framework to generate an overall MQM score (rated on a 0-100 scale). Secondary outcomes included a general preference rating and error rates for types of translation errors.
Results
This study included 20 source files of pediatric patient instructions. Equivalence testing showed no significant difference in translation quality between GPT-4o and human translations, with a mean difference of 1.6 points (90% CI, 0.7-2.5), falling within a predefined equivalence margin of plus or minus 5 MQM points. The LLM yielded fewer mistranslation errors, and a mean (SE) of 52% (6%) of professional translator ratings preferred the LLM translations. Conclusions and Relevance In this cross-sectional study, GPT-4o generated Spanish translations of pediatric patient instructions that were comparable in quality to those by professional human translators as evaluated using a standardized framework. While human review of LLM translation remains essential in health care, these findings suggest that GPT-4o could reduce the translation workload for Spanish, potentially freeing resources to support languages of lesser diffusion.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Division of General Pediatrics, Boston Children's Hospital, Boston, Massachusetts.
- Department of Pediatrics, Harvard Medical School, Boston, Massachusetts.
- School of Applied Language and Intercultural Studies, Dublin City University, Dublin, Ireland.
- Innovation and Digital Health Accelerator, Boston Children's Hospital, Boston, Massachusetts.
- Interpreter Services, Boston Children's Hospital, Boston, Massachusetts.
- Department of Neonatology, Beth Israel Deaconess Medical Center, Boston, Massachusetts.
Förderung
- NLM NIH HHS
Zitieren
APA
Ray, M., Kats, D. J., Moorkens, J., Rai, D., Shaar, N., Quinones, D., Vermeulen, A., Mateo, C. M., Brewster, R. C. L., Khan, A., Rader, B., Brownstein, J. S., & Hron, J. D. (2025). Evaluating a Large Language Model in Translating Patient Instructions to Spanish Using a Standardized Framework. JAMA Pediatrics, 179(9), 1026. https://doi.org/10.1001/jamapediatrics.2025.1729
Vancouver
Ray M, Kats DJ, Moorkens J, Rai D, Shaar N, Quinones D, et al. Evaluating a Large Language Model in Translating Patient Instructions to Spanish Using a Standardized Framework. JAMA Pediatrics. 2025;179(9):1026. doi:10.1001/jamapediatrics.2025.1729
BibTeX
@article{Ray2025,
author = {Ray, Mondira and Kats, Daniel J and Moorkens, Joss and Rai, Dinesh and Shaar, Nate and Quinones, Diane and Vermeulen, Alejandro and Mateo, Camila M and Brewster, Ryan C L and Khan, Alisa and Rader, Benjamin and Brownstein, John S and Hron, Jonathan D},
title = {Evaluating a Large Language Model in Translating Patient Instructions to Spanish Using a Standardized Framework},
journal = {JAMA Pediatrics},
year = {2025},
volume = {179},
number = {9},
pages = {1026},
doi = {10.1001/jamapediatrics.2025.1729}
}
EUVIMED