GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals
Journal of Computer Science and Frontier Technologies · 10.06.2026 · 3 (3) · 11
DOI: 10.63313/jcsft.9083 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Computer Science and Frontier Technologies, online erschienen am 10.06.2026. Verfasst von Qianyun Lin und 1 weiteren Autor:innen. Die Arbeit nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).
Abstract (Originalsprache)
Accurate and efficient diagnosis of gastrointestinal (GI) diseases is essential for timely clinical decision-making, yet traditional diagnostic workflows rely heavily on manual interpretation of endoscopic images and disconnected physiological measurements. These limitations often lead to variability in diagnosis and increased clinician workload. To address these challenges, we propose GIMeT (GastroIntestinal Multimodal Transformer), a novel deep learning framework that integrates endoscopic imaging, physiological time-series data, and optional clinical text information for automated GI disease classification and lesion segmentation. GIMeT consists of three modality-specific encoders—a CNN-based image encoder, a temporal encoder for physiological signals, and a BERT-based text encoder—followed by a Cross-Modal Transformer (CMT) to achieve fine-grained feature alignment and joint representation learning. A FiLM-conditioned U-Net decoder further enhances pixel-level lesion segmentation using multimodal cues. We evaluate GIMeT on a multi-center dataset comprising endoscopic images, electro-physiological measurements, and symptom descriptions. Experimental results show that GIMeT consistently surpasses all unimodal and multimodal baselines across both classification and segmentation tasks. The model achieves improvements of approximately 3-4% in accuracy and F1-score, and a 3% gain in lesion Dice coefficient. Ablation studies further validate the contribution of multimodal fusion and cross-modal attention, demonstrating GIMeT’s robustness and strong potential for computer-assisted gastrointestinal diagnosis.
Zitieren
APA
Lin, Q., & Cheung, A. (2026). GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals. Journal of Computer Science and Frontier Technologies, 3(3), 11. https://doi.org/10.63313/jcsft.9083
Vancouver
Lin Q, Cheung A. GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals. Journal of Computer Science and Frontier Technologies. 2026;3(3):11. doi:10.63313/jcsft.9083
BibTeX
@article{Lin2026,
author = {Lin, Qianyun and Cheung, Annie},
title = {GIMeT: A Multimodal Transformer Framework for Automated Gastrointestinal Disease Diagnosis Using Endoscopic Images and Physiological Signals},
journal = {Journal of Computer Science and Frontier Technologies},
year = {2026},
volume = {3},
number = {3},
pages = {11},
doi = {10.63313/jcsft.9083}
}
EUVIMED