EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight

Hossein Rouhizadeh, Abiram Sandralegar, Anthony Yazdani, Weibo Feng, Oren Schreier, Yonnou Ahn-Kim, Assiya Sirbal, Valentino Pirelli, Rui Yang, Lukas Sveikata, Elena Tessitore, Nan Liu iD und 2 weitere

Journal of the American Medical Informatics Association · 01.07.2026 · 33 (7) · 1303-1313

DOI: 10.1093/jamia/ocag070 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 08.05.2026. Verfasst von Hossein Rouhizadeh und 13 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Geneva, 1202, Switzerland. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Multilingualism, Documentation. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

The automation of medical report generation using large language models (LLMs) could significantly reduce physicians’ documentation burden while enhancing healthcare efficiency. However, the misuse of generative artificial intelligence in medical reporting can lead to important safety risks for patients. We addressed 2 questions: (1) What is the quality of medical reports generated by LLMs in English and French? and (2) Can we distinguish between human-written and LLM-generated medical reports? Materials and methods We evaluated the quality of reports generated by several multilingual, open-weight LLMs using text similarity metrics on 4212 medical reports in English and French across multiple specialties. A bilingual expert panel of certified physicians (n = 4) and medical residents (n = 5) scored accuracy, fluency, and completeness of generated reports using a 1-5 Likert scale. Experts also completed a Turing-like test, blindly identifying reports as human or machine-generated.

Results

Phi-4 achieved the best overall performance (ROUGE-1: 0.70, BERTScore: 0.83). Expert evaluation confirmed high-quality reports in both languages (overall 4.6/5.0). Medical experts performed better than chance but struggled to differentiate human versus machine reports (accuracy: 0.60). Automatic classifiers showed strong performance (accuracy: 0.98).

Discussion

The high quality of LLM-generated reports supports their potential to enhance healthcare efficiency in multilingual settings. However, the discrepancy between human detection difficulty and automated detection success reveals inherent limitations in relying solely on human oversight for quality assurance and misuse prevention.

Conclusions

Deployment of LLMs for medical reporting requires combining automated detection tools with human expertise to ensure patient safety. Dataset and code: https://github.com/ds4dh/medical_report_generation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsElectronic Health RecordsMultilingualismDocumentationHumansGenerative Artificial Intelligence

Stichwörter der Autor:innen

automated report generationelectronic health records (EHRs)large language models (LLMs)medical documentationmultilingual medical reports

Autor:innen und Institutionen

  • Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Geneva, 1202, Switzerland.
  • Division of Neurosurgery, Geneva University Hospitals, Geneva, 1211, Switzerland.
  • Faculty of Medicine, University of Geneva, Geneva, 1211, Switzerland.
  • Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, 169857, Singapore.
  • Division of Neurology, Department of Clinical Neurosciences, Geneva University Hospitals and Faculty of Medicine, Geneva, 1211, Switzerland.
  • Cardiology Division, Internal Medicine Department, Geneva University Hospitals, Geneva, 1211, Switzerland.

ORCID-Profile: Nan Liu, Douglas Teodoro

Zitieren

APA

Rouhizadeh, H., Sandralegar, A., Yazdani, A., Feng, W., Schreier, O., Ahn-Kim, Y., Sirbal, A., Pirelli, V., Yang, R., Sveikata, L., Tessitore, E., Liu, N., Bijlenga, P., & Teodoro, D. (2026). The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight. Journal of the American Medical Informatics Association, 33(7), 1303-1313. https://doi.org/10.1093/jamia/ocag070

Vancouver

Rouhizadeh H, Sandralegar A, Yazdani A, Feng W, Schreier O, Ahn-Kim Y, et al. The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight. Journal of the American Medical Informatics Association. 2026;33(7):1303-1313. doi:10.1093/jamia/ocag070

BibTeX
@article{Rouhizadeh2026,
  author = {Rouhizadeh, Hossein and Sandralegar, Abiram and Yazdani, Anthony and Feng, Weibo and Schreier, Oren and Ahn-Kim, Yonnou and Sirbal, Assiya and Pirelli, Valentino and Yang, Rui and Sveikata, Lukas and Tessitore, Elena and Liu, Nan and Bijlenga, Philippe and Teodoro, Douglas},
  title = {The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  volume = {33},
  number = {7},
  pages = {1303--1313},
  doi = {10.1093/jamia/ocag070}
}
Kostenlos starten →