EUVIMEDEUVIMED entdecken
Crossref

Token-Mixer: Bind Image and Text in One Embedding Space for Medical Image Reporting

Yan Yang, Jun Yu, Zhenqi Fu, Ke Zhang, Ting Yu, Xianyun Wang, Hanliang Jiang, Junhui Lv, Qingming Huang, Weidong Han

IEEE Transactions on Medical Imaging · 2024-11 · 43 (11) · 4017-4028

DOI: 10.1109/tmi.2024.3412402 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2024-11. Verfasst von Yan Yang und 9 weiteren Autor:innen. Hauptthemen laut MEDLINE: Algorithms. Die Arbeit wurde bisher 42-mal zitiert (Crossref) und nennt 62 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Zhejiang Provincial Natural Science Foundation of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Medical image reporting focused on automatically generating the diagnostic reports from medical images has garnered growing research attention. In this task, learning cross-modal alignment between images and reports is crucial. However, the exposure bias problem in autoregressive text generation poses a notable challenge, as the model is optimized by a word-level loss function using the teacher-forcing strategy. To this end, we propose a novel Token-Mixer framework that learns to bind image and text in one embedding space for medical image reporting. Concretely, Token-Mixer enhances the cross-modal alignment by matching image-to-text generation with text-to-text generation that suffers less from exposure bias. The framework contains an image encoder, a text encoder and a text decoder. In training, images and paired reports are first encoded into image tokens and text tokens, and these tokens are randomly mixed to form the mixed tokens. Then, the text decoder accepts image tokens, text tokens or mixed tokens as prompt tokens and conducts text generation for network optimization. Furthermore, we introduce a tailored text decoder and an alternative training strategy that well integrate with our Token-Mixer framework. Extensive experiments across three publicly available datasets demonstrate Token-Mixer successfully enhances the image-text alignment and thereby attains a state-of-the-art performance. Related codes are available at https://github.com/yangyan22/Token-Mixer.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AlgorithmsHumansDiagnostic ImagingImage Processing, Computer-AssistedDatabases, Factual

Förderung

  • National Natural Science Foundation of China
  • Zhejiang Provincial Natural Science Foundation of China

Zitieren

APA

Yang, Y., Yu, J., Fu, Z., Zhang, K., Yu, T., Wang, X., Jiang, H., Lv, J., Huang, Q., & Han, W. (2024). Token-Mixer: Bind Image and Text in One Embedding Space for Medical Image Reporting. IEEE Transactions on Medical Imaging, 43(11), 4017-4028. https://doi.org/10.1109/tmi.2024.3412402

Vancouver

Yang Y, Yu J, Fu Z, Zhang K, Yu T, Wang X, et al. Token-Mixer: Bind Image and Text in One Embedding Space for Medical Image Reporting. IEEE Transactions on Medical Imaging. 2024;43(11):4017-4028. doi:10.1109/tmi.2024.3412402

BibTeX
@article{Yang2024,
  author = {Yang, Yan and Yu, Jun and Fu, Zhenqi and Zhang, Ke and Yu, Ting and Wang, Xianyun and Jiang, Hanliang and Lv, Junhui and Huang, Qingming and Han, Weidong},
  title = {Token-Mixer: Bind Image and Text in One Embedding Space for Medical Image Reporting},
  journal = {IEEE Transactions on Medical Imaging},
  year = {2024},
  volume = {43},
  number = {11},
  pages = {4017--4028},
  doi = {10.1109/tmi.2024.3412402}
}
Kostenlos starten →