EUVIMEDEUVIMED entdecken
Open Access · CC BY-SA 4.0

GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals

Qianyun Lin, Annie Cheung

Journal of Computer Science and Frontier Technologies · 10.06.2026 · 3 (3) · 11

DOI: 10.63313/jcsft.9083 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Computer Science and Frontier Technologies, online erschienen am 10.06.2026. Verfasst von Qianyun Lin und 1 weiteren Autor:innen. Die Arbeit nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Accurate and efficient diagnosis of gastrointestinal (GI) diseases is essential for timely clinical decision-making, yet traditional diagnostic workflows rely heavily on manual interpretation of endoscopic images and disconnected physiological measurements. These limitations often lead to variability in diagnosis and increased clinician workload. To address these challenges, we propose GIMeT (GastroIntestinal Multimodal Transformer), a novel deep learning framework that integrates endoscopic imaging, physiological time-series data, and optional clinical text information for automated GI disease classification and lesion segmentation. GIMeT consists of three modality-specific encoders—a CNN-based image encoder, a temporal encoder for physiological signals, and a BERT-based text encoder—followed by a Cross-Modal Transformer (CMT) to achieve fine-grained feature alignment and joint representation learning. A FiLM-conditioned U-Net decoder further enhances pixel-level lesion segmentation using multimodal cues. We evaluate GIMeT on a multi-center dataset comprising endoscopic images, electro-physiological measurements, and symptom descriptions. Experimental results show that GIMeT consistently surpasses all unimodal and multimodal baselines across both classification and segmentation tasks. The model achieves improvements of approximately 3-4% in accuracy and F1-score, and a 3% gain in lesion Dice coefficient. Ablation studies further validate the contribution of multimodal fusion and cross-modal attention, demonstrating GIMeT’s robustness and strong potential for computer-assisted gastrointestinal diagnosis.

Zitieren

APA

Lin, Q., & Cheung, A. (2026). GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals. Journal of Computer Science and Frontier Technologies, 3(3), 11. https://doi.org/10.63313/jcsft.9083

Vancouver

Lin Q, Cheung A. GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals. Journal of Computer Science and Frontier Technologies. 2026;3(3):11. doi:10.63313/jcsft.9083

BibTeX
@article{Lin2026,
  author = {Lin, Qianyun and Cheung, Annie},
  title = {GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals},
  journal = {Journal of Computer Science and Frontier Technologies},
  year = {2026},
  volume = {3},
  number = {3},
  pages = {11},
  doi = {10.63313/jcsft.9083}
}
Kostenlos starten →