The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight
Journal of the American Medical Informatics Association · 01.07.2026 · 33 (7) · 1303-1313
DOI: 10.1093/jamia/ocag070 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 08.05.2026. Verfasst von Hossein Rouhizadeh und 13 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Geneva, 1202, Switzerland. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Multilingualism, Documentation. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objectives
The automation of medical report generation using large language models (LLMs) could significantly reduce physicians’ documentation burden while enhancing healthcare efficiency. However, the misuse of generative artificial intelligence in medical reporting can lead to important safety risks for patients. We addressed 2 questions: (1) What is the quality of medical reports generated by LLMs in English and French? and (2) Can we distinguish between human-written and LLM-generated medical reports? Materials and methods We evaluated the quality of reports generated by several multilingual, open-weight LLMs using text similarity metrics on 4212 medical reports in English and French across multiple specialties. A bilingual expert panel of certified physicians (n = 4) and medical residents (n = 5) scored accuracy, fluency, and completeness of generated reports using a 1-5 Likert scale. Experts also completed a Turing-like test, blindly identifying reports as human or machine-generated.
Results
Phi-4 achieved the best overall performance (ROUGE-1: 0.70, BERTScore: 0.83). Expert evaluation confirmed high-quality reports in both languages (overall 4.6/5.0). Medical experts performed better than chance but struggled to differentiate human versus machine reports (accuracy: 0.60). Automatic classifiers showed strong performance (accuracy: 0.98).
Discussion
The high quality of LLM-generated reports supports their potential to enhance healthcare efficiency in multilingual settings. However, the discrepancy between human detection difficulty and automated detection success reveals inherent limitations in relying solely on human oversight for quality assurance and misuse prevention.
Conclusions
Deployment of LLMs for medical reporting requires combining automated detection tools with human expertise to ensure patient safety. Dataset and code: https://github.com/ds4dh/medical_report_generation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Geneva, 1202, Switzerland.
- Division of Neurosurgery, Geneva University Hospitals, Geneva, 1211, Switzerland.
- Faculty of Medicine, University of Geneva, Geneva, 1211, Switzerland.
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, 169857, Singapore.
- Division of Neurology, Department of Clinical Neurosciences, Geneva University Hospitals and Faculty of Medicine, Geneva, 1211, Switzerland.
- Cardiology Division, Internal Medicine Department, Geneva University Hospitals, Geneva, 1211, Switzerland.
ORCID-Profile: Nan Liu, Douglas Teodoro
Zitieren
APA
Rouhizadeh, H., Sandralegar, A., Yazdani, A., Feng, W., Schreier, O., Ahn-Kim, Y., Sirbal, A., Pirelli, V., Yang, R., Sveikata, L., Tessitore, E., Liu, N., Bijlenga, P., & Teodoro, D. (2026). The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight. Journal of the American Medical Informatics Association, 33(7), 1303-1313. https://doi.org/10.1093/jamia/ocag070
Vancouver
Rouhizadeh H, Sandralegar A, Yazdani A, Feng W, Schreier O, Ahn-Kim Y, et al. The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight. Journal of the American Medical Informatics Association. 2026;33(7):1303-1313. doi:10.1093/jamia/ocag070
BibTeX
@article{Rouhizadeh2026,
author = {Rouhizadeh, Hossein and Sandralegar, Abiram and Yazdani, Anthony and Feng, Weibo and Schreier, Oren and Ahn-Kim, Yonnou and Sirbal, Assiya and Pirelli, Valentino and Yang, Rui and Sveikata, Lukas and Tessitore, Elena and Liu, Nan and Bijlenga, Philippe and Teodoro, Douglas},
title = {The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
volume = {33},
number = {7},
pages = {1303--1313},
doi = {10.1093/jamia/ocag070}
}
EUVIMED