EUVIMEDOpdag EUVIMED
ValidierungsstudieOpen access

Identification of Syndrome Types in Patients With Pancreatic Cancer From Free Text in Electronic Medical Records: Model Development and Validation

He Ba iD, Haojie Du iD, Chienshan Cheng iD, Yuan Zhang iD, Linjie Ruan iD, Zhen Chen iD

JMIR Formative Research · 2025

DOI: 10.2196/70602 ↗

At a glance

Validierungsstudie in JMIR Formative Research, 2025. Written by He Ba and 5 co-authors. Institution: Qingdao Institute, Department of Integrative Oncology, Fudan University Shanghai Cancer Center, Qingdao, China. Major MEDLINE topics: Pancreatic Neoplasms, Medicine, Chinese Traditional, Electronic Health Records. Cited 3 times so far (Crossref); 62 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Syndrome differentiation is crucial in traditional Chinese medicine (TCM) diagnosis and treatment, but it heavily relies on expert experience, limiting systematic standardization.

Objective

This study developed and validated a BERT (bidirectional encoder representations from transformers)–based model, the traditional Chinese medicine pancreatic cancer syndrome differentiation bidirectional encoder representations from transformers (TCMPCSD-BERT), using in-house pancreatic cancer medical records, to digitalize expert knowledge and support standardized syndrome differentiation in TCM.

Methods

A retrospective dataset of pancreatic cancer cases (2011-2024) from Fudan University Shanghai Cancer Center was annotated into 4 TCM syndrome types by 2 experts (Cohen κ=0.913). The proposed TCMPCSD-BERT model was compared with conventional models (long short-term memory and text convolutional neural network) embedded in TCM diagnostic tools and with large language models (LLMs; ChatGPT-4o, Kimi, Ernie Bot 4.0 Turbo, and Zhipu Qingyan) under a prompt engineering framework. Performance evaluation on in-house data was supplemented with attention visualizations and integrated gradients analyses for interpretability. The McNemar test assessed classification accuracy differences, while bootstrap 95% CIs quantified statistical uncertainty and stability. The Welch t test (2-tailed) was used to evaluate mean differences between TCMPCSD-BERT and the comparator models.

Results

Among 6830 records, case counts were damp-heat syndrome (n=1694), spleen-deficiency syndrome (n=1185), damp-heat with spleen-deficiency syndrome (n=1178), and others (n=2773). On the test set, McNemar test showed significantly higher accuracy for TCMPCSD-BERT than the 3 baseline models and generally better performance than LLMs. In all comparisons, TCMPCSD-BERT achieved higher mean macroprecision, macrorecall, macro–F1-score, and accuracy, with nonoverlapping 95% bootstrap CIs and significant Welch t test results (P

Subjects

MeSH terms (MEDLINE)

Pancreatic NeoplasmsMedicine, Chinese TraditionalElectronic Health RecordsHumansRetrospective StudiesDiagnosis, DifferentialMaleSyndromeNeural Networks, ComputerFemale

Author keywords

BERTbidirectional encoder representations from transformersnatural language processingpancreatic cancersyndrome differentiationtraditional Chinese medicine

Authors and institutions

  • Qingdao Institute, Department of Integrative Oncology, Fudan University Shanghai Cancer Center, Qingdao, China.
  • Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China.
  • Key Laboratory of Surface & Interface Science of Polymer Materials of Zhejiang Province, School of Chemistry and Chemical Engineering, Zhejiang Sci-Tech University, Hangzhou, China.
  • Shengzhou Innovation Research Institute of Zhejiang Sci-Tech University, Shengzhou, China.
  • Department of Integrative Oncology, Fudan University Shanghai Cancer Center, Shanghai, China.

ORCID profiles: He Ba, Haojie Du, Chienshan Cheng, Yuan Zhang, Linjie Ruan, Zhen Chen

Cite

APA

Ba, H., Du, H., Cheng, C., Zhang, Y., Ruan, L., & Chen, Z. (2025). Identification of Syndrome Types in Patients With Pancreatic Cancer From Free Text in Electronic Medical Records: Model Development and Validation. JMIR Formative Research. https://doi.org/10.2196/70602

Vancouver

Ba H, Du H, Cheng C, Zhang Y, Ruan L, Chen Z. Identification of Syndrome Types in Patients With Pancreatic Cancer From Free Text in Electronic Medical Records: Model Development and Validation. JMIR Formative Research. 2025. doi:10.2196/70602

BibTeX
@article{Ba2025,
  author = {Ba, He and Du, Haojie and Cheng, Chienshan and Zhang, Yuan and Ruan, Linjie and Chen, Zhen},
  title = {Identification of Syndrome Types in Patients With Pancreatic Cancer From Free Text in Electronic Medical Records: Model Development and Validation},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/70602}
}
Start for free →