EUVIMEDSkopri EUVIMED
Open access · CC BY-SA 4.0

GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals

Qianyun Lin, Annie Cheung

Journal of Computer Science and Frontier Technologies · 2026-06-10 · 3 (3) · 11

DOI: 10.63313/jcsft.9083 ↗

At a glance

Journal article in Journal of Computer Science and Frontier Technologies, published online on 2026-06-10. Written by Qianyun Lin and 1 co-authors. 20 references. The full text is freely available (open access, CC BY-SA 4.0).

AstrattDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Accurate and efficient diagnosis of gastrointestinal (GI) diseases is essential for timely clinical decision-making, yet traditional diagnostic workflows rely heavily on manual interpretation of endoscopic images and disconnected physiological measurements. These limitations often lead to variability in diagnosis and increased clinician workload. To address these challenges, we propose GIMeT (GastroIntestinal Multimodal Transformer), a novel deep learning framework that integrates endoscopic imaging, physiological time-series data, and optional clinical text information for automated GI disease classification and lesion segmentation. GIMeT consists of three modality-specific encoders—a CNN-based image encoder, a temporal encoder for physiological signals, and a BERT-based text encoder—followed by a Cross-Modal Transformer (CMT) to achieve fine-grained feature alignment and joint representation learning. A FiLM-conditioned U-Net decoder further enhances pixel-level lesion segmentation using multimodal cues. We evaluate GIMeT on a multi-center dataset comprising endoscopic images, electro-physiological measurements, and symptom descriptions. Experimental results show that GIMeT consistently surpasses all unimodal and multimodal baselines across both classification and segmentation tasks. The model achieves improvements of approximately 3-4% in accuracy and F1-score, and a 3% gain in lesion Dice coefficient. Ablation studies further validate the contribution of multimodal fusion and cross-modal attention, demonstrating GIMeT’s robustness and strong potential for computer-assisted gastrointestinal diagnosis.

Cite

APA

Lin, Q., & Cheung, A. (2026). GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals. Journal of Computer Science and Frontier Technologies, 3(3), 11. https://doi.org/10.63313/jcsft.9083

Vancouver

Lin Q, Cheung A. GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals. Journal of Computer Science and Frontier Technologies. 2026;3(3):11. doi:10.63313/jcsft.9083

BibTeX
@article{Lin2026,
  author = {Lin, Qianyun and Cheung, Annie},
  title = {GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals},
  journal = {Journal of Computer Science and Frontier Technologies},
  year = {2026},
  volume = {3},
  number = {3},
  pages = {11},
  doi = {10.63313/jcsft.9083}
}
Start for free →