EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis

Wataru Arihisa iD, Tomohiro Nishiyama iD, Shoko Wakamiya iD, Eiji Aramaki iD

JMIR Formative Research · 2026

DOI: 10.2196/85671 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Formative Research, 2026. Verfasst von Wataru Arihisa und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, 6…. Hauptthemen laut MEDLINE: Documentation, Large Language Models, Physicians, Psychiatry. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

With the rapid adoption of large language models (LLMs) in clinical documentation, it is unclear whether LLMs can faithfully reproduce specialty-specific writing styles and clinically meaningful documentation patterns observed in expert notes, particularly in psychiatry.

Objective

This study aims to systematically compare the narrative styles of human physicians and LLMs when documenting identical psychiatric cases and to evaluate the extent to which LLMs replicate specialty-specific documentation patterns.

Methods

We constructed 2 standardized outpatient scenarios in Japanese (major depressive disorder and schizophrenia) and collected 134 initial notes in Japanese authored by psychiatrists and internists, alongside notes generated by 4 LLMs simulating each specialty. We conducted lexical, syntactic, semantic, and topic-level analyses using Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation–Longest Common Subsequence (ROUGE-L), BERTScore, and Translation Edit Rate (TER), complemented by redundancy metrics and medical term variation analyses.

Results

LLM-generated notes were significantly longer, more repetitive, and lexically less diverse than human-authored notes. TER-based clustering revealed a uniform, template-like writing style in LLMs, diverging from the flexible, context-sensitive style of physicians. Topic modeling suggested that LLM-generated notes tended to rely on more abstract and generalized expressions, with less variation in the distribution and emphasis of documented clinical information.

Conclusions

LLMs can mimic surface-level stylistic features but fall short in reproducing nuanced, context-dependent, diagnostically relevant content typical of expert clinical documentation. Future clinical use will require careful prompt design or fine-tuning to ensure narrative depth, lexical diversity, and clinical relevance.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DocumentationLarge Language ModelsPhysiciansPsychiatryHumansJapan

Stichwörter der Autor:innen

clinical documentationlarge language modelsmedical informaticsnatural language processingpsychiatry

Autor:innen und Institutionen

  • Department of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, 630-0192, Japan, 81 0743-72-5111.

ORCID-Profile: Wataru Arihisa, Tomohiro Nishiyama, Shoko Wakamiya, Eiji Aramaki

Zitieren

APA

Arihisa, W., Nishiyama, T., Wakamiya, S., & Aramaki, E. (2026). Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis. JMIR Formative Research. https://doi.org/10.2196/85671

Vancouver

Arihisa W, Nishiyama T, Wakamiya S, Aramaki E. Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis. JMIR Formative Research. 2026. doi:10.2196/85671

BibTeX
@article{Arihisa2026,
  author = {Arihisa, Wataru and Nishiyama, Tomohiro and Wakamiya, Shoko and Aramaki, Eiji},
  title = {Comparative Analysis of Japanese Clinical Note Styles Between Physicians and Large Language Models Using Identical Psychiatric Cases: Quantitative Text Analysis},
  journal = {JMIR Formative Research},
  year = {2026},
  doi = {10.2196/85671}
}
Kostenlos starten →