EUVIMEDObjevte EUVIMED
VergleichsstudieOpen access

Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis

Wataru Arihisa iD, Tomohiro Nishiyama iD, Shoko Wakamiya iD, Eiji Aramaki iD

JMIR Formative Research · 2026

DOI: 10.2196/85671 ↗

At a glance

Vergleichsstudie in JMIR Formative Research, 2026. Written by Wataru Arihisa and 3 co-authors. Institution: Department of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, 6…. Major MEDLINE topics: Documentation, Large Language Models, Physicians, Psychiatry. Cited 2 times so far (Crossref); 42 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

With the rapid adoption of large language models (LLMs) in clinical documentation, it is unclear whether LLMs can faithfully reproduce specialty-specific writing styles and clinically meaningful documentation patterns observed in expert notes, particularly in psychiatry.

Objective

This study aims to systematically compare the narrative styles of human physicians and LLMs when documenting identical psychiatric cases and to evaluate the extent to which LLMs replicate specialty-specific documentation patterns.

Methods

We constructed 2 standardized outpatient scenarios in Japanese (major depressive disorder and schizophrenia) and collected 134 initial notes in Japanese authored by psychiatrists and internists, alongside notes generated by 4 LLMs simulating each specialty. We conducted lexical, syntactic, semantic, and topic-level analyses using Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation–Longest Common Subsequence (ROUGE-L), BERTScore, and Translation Edit Rate (TER), complemented by redundancy metrics and medical term variation analyses.

Results

LLM-generated notes were significantly longer, more repetitive, and lexically less diverse than human-authored notes. TER-based clustering revealed a uniform, template-like writing style in LLMs, diverging from the flexible, context-sensitive style of physicians. Topic modeling suggested that LLM-generated notes tended to rely on more abstract and generalized expressions, with less variation in the distribution and emphasis of documented clinical information.

Conclusions

LLMs can mimic surface-level stylistic features but fall short in reproducing nuanced, context-dependent, diagnostically relevant content typical of expert clinical documentation. Future clinical use will require careful prompt design or fine-tuning to ensure narrative depth, lexical diversity, and clinical relevance.

Subjects

MeSH terms (MEDLINE)

DocumentationLarge Language ModelsPhysiciansPsychiatryHumansJapan

Author keywords

clinical documentationlarge language modelsmedical informaticsnatural language processingpsychiatry

Authors and institutions

  • Department of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, 630-0192, Japan, 81 0743-72-5111.

ORCID profiles: Wataru Arihisa, Tomohiro Nishiyama, Shoko Wakamiya, Eiji Aramaki

Cite

APA

Arihisa, W., Nishiyama, T., Wakamiya, S., & Aramaki, E. (2026). Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis. JMIR Formative Research. https://doi.org/10.2196/85671

Vancouver

Arihisa W, Nishiyama T, Wakamiya S, Aramaki E. Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis. JMIR Formative Research. 2026. doi:10.2196/85671

BibTeX
@article{Arihisa2026,
  author = {Arihisa, Wataru and Nishiyama, Tomohiro and Wakamiya, Shoko and Aramaki, Eiji},
  title = {Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis},
  journal = {JMIR Formative Research},
  year = {2026},
  doi = {10.2196/85671}
}
Start for free →