Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis
JMIR Formative Research · 2026
DOI: 10.2196/85671 ↗At a glance
Vergleichsstudie in JMIR Formative Research, 2026. Written by Wataru Arihisa and 3 co-authors. Institution: Department of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, 6…. Major MEDLINE topics: Documentation, Large Language Models, Physicians, Psychiatry. Cited 2 times so far (Crossref); 42 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
With the rapid adoption of large language models (LLMs) in clinical documentation, it is unclear whether LLMs can faithfully reproduce specialty-specific writing styles and clinically meaningful documentation patterns observed in expert notes, particularly in psychiatry.
Objective
This study aims to systematically compare the narrative styles of human physicians and LLMs when documenting identical psychiatric cases and to evaluate the extent to which LLMs replicate specialty-specific documentation patterns.
Methods
We constructed 2 standardized outpatient scenarios in Japanese (major depressive disorder and schizophrenia) and collected 134 initial notes in Japanese authored by psychiatrists and internists, alongside notes generated by 4 LLMs simulating each specialty. We conducted lexical, syntactic, semantic, and topic-level analyses using Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation–Longest Common Subsequence (ROUGE-L), BERTScore, and Translation Edit Rate (TER), complemented by redundancy metrics and medical term variation analyses.
Results
LLM-generated notes were significantly longer, more repetitive, and lexically less diverse than human-authored notes. TER-based clustering revealed a uniform, template-like writing style in LLMs, diverging from the flexible, context-sensitive style of physicians. Topic modeling suggested that LLM-generated notes tended to rely on more abstract and generalized expressions, with less variation in the distribution and emphasis of documented clinical information.
Conclusions
LLMs can mimic surface-level stylistic features but fall short in reproducing nuanced, context-dependent, diagnostically relevant content typical of expert clinical documentation. Future clinical use will require careful prompt design or fine-tuning to ensure narrative depth, lexical diversity, and clinical relevance.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, 630-0192, Japan, 81 0743-72-5111.
ORCID profiles: Wataru Arihisa, Tomohiro Nishiyama, Shoko Wakamiya, Eiji Aramaki
Cite
APA
Arihisa, W., Nishiyama, T., Wakamiya, S., & Aramaki, E. (2026). Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis. JMIR Formative Research. https://doi.org/10.2196/85671
Vancouver
Arihisa W, Nishiyama T, Wakamiya S, Aramaki E. Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis. JMIR Formative Research. 2026. doi:10.2196/85671
BibTeX
@article{Arihisa2026,
author = {Arihisa, Wataru and Nishiyama, Tomohiro and Wakamiya, Shoko and Aramaki, Eiji},
title = {Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis},
journal = {JMIR Formative Research},
year = {2026},
doi = {10.2196/85671}
}
EUVIMED