Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality
PLOS Digital Health · 26.10.2026 · 5 (10) · e0001799
DOI: 10.1371/journal.pdig.0001799 ↗Kurz gefasst
Randomisierte kontrollierte Studie in der Zeitschrift PLOS Digital Health, online erschienen am 26.10.2026. Verfasst von Tomohiro Takayama und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Ophthalmology and Visual Sciences, Kyoto University Graduate School of Medicine, Kyoto, Japan. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) have been investigated for clinical documentation, with concerns about hallucinations and factual errors. Clinician review and revision of LLM-generated drafts are considered essential; however, the impact of such a workflow on both documentation time and quality remains unknown. This single-center, parallel-group, prospective, randomized, open-label, blinded-endpoint (PROBE) pilot trial evaluated whether a clinician-in-the-loop workflow improves clinical documentation time and quality compared with clinician-only drafting. From February 18 to March 14, 2025, at the Department of Ophthalmology, Kyoto University Hospital, 21 physicians were randomized, and 17 completed the intervention. Participants were randomized to either the Clinician-in-the-loop group or the Clinician-only group. Participants created discharge summaries and referrals for six simulated patient records. The Clinician-in-the-loop group reviewed and edited LLM-generated drafts, whereas the Clinician-only group drafted documents from scratch using matched templates. Unedited LLM drafts formed the LLM-only group. Seventeen physicians submitted 48 discharge summaries and 48 discharge referrals in the Clinician-in-the-loop group, 54 of each document type in the Clinician-only group, and 48 of each in the LLM-only group. Clinician-in-the-loop showed a shorter creation time for summaries (β = −59.4 seconds; 95% CI, −150.6 to 31.7; P = .18) and a longer creation time for referrals (β = 52.9 seconds; 95% CI, −7.0 to 112.9; P = .08) than Clinician-only, although neither difference was statistically significant. In most expert-rated quality domains, the clinician-in-the-loop workflow outperformed clinician-only drafting. LLM-only drafts were fastest but had the lowest quality. In conclusion, a clinician-in-the-loop approach generally improved document quality and, although the primary analysis showed no statistically significant difference in documentation time, suggested potential time savings for certain document types. These findings highlight the importance of active clinician supervision and support further evaluation of clinician-in-the-loop workflows in real-world clinical settings. Trial Registration ClinicalTrials.gov Identifier NCT07187050 (registered on September 18, 2025).
Autor:innen und Institutionen
- Department of Ophthalmology and Visual Sciences, Kyoto University Graduate School of Medicine, Kyoto, Japan.
- Kyoto University Hospital, Kyoto, Japan.
- Fitting Cloud Inc., Kyoto, Japan.
- Center for Innovative Research and Education in Data Science, Institute for Liberal Arts and Sciences, Kyoto University, Kyoto, Japan.
- Japanese Society of Artificial Intelligence in Ophthalmology, Chuoku, Tokyo, Japan.
- Department of Ophthalmology, Faculty of Medicine, University of Tsukuba, Ibaraki, Japan.
ORCID-Profile: Tomohiro Takayama, Hiroshi Tamura, Luciano Henrique de Oliveira Santos, Tetsuro Oshika, Tomohiro Kuroda, Masahiro Miyake
Zitieren
APA
Takayama, T., Sado, K., Suda, K., Tamura, H., Ueda-Arakawa, N., Ishihara, K., Ueda, Y., Okamoto, K., Santos, L. H. O., Oshika, T., Kuroda, T., Tsujikawa, A., & Miyake, M. (2026). Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality. PLOS Digital Health, 5(10), e0001799. https://doi.org/10.1371/journal.pdig.0001799
Vancouver
Takayama T, Sado K, Suda K, Tamura H, Ueda-Arakawa N, Ishihara K, et al. Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality. PLOS Digital Health. 2026;5(10):e0001799. doi:10.1371/journal.pdig.0001799
BibTeX
@article{Takayama2026,
author = {Takayama, Tomohiro and Sado, Keina and Suda, Kenji and Tamura, Hiroshi and Ueda-Arakawa, Naoko and Ishihara, Kenji and Ueda, Yuta and Okamoto, Kazuya and Santos, Luciano Henrique de Oliveira and Oshika, Tetsuro and Kuroda, Tomohiro and Tsujikawa, Akitaka and Miyake, Masahiro},
title = {Evaluating an LLM-assisted workflow for clinical documentation: A pilot randomized controlled trial on time and quality},
journal = {PLOS Digital Health},
year = {2026},
volume = {5},
number = {10},
pages = {e0001799},
doi = {10.1371/journal.pdig.0001799}
}
EUVIMED