EUVIMEDEUVIMED entdecken
Crossref

A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models

Weixin Ye, Wei Wang, Yahui Liu, Yue Song, Bin Ren, Wei Bi, Rita Cucchiara, Nicu Sebe

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-02 · 48 (2) · 1873-1887

DOI: 10.1109/tpami.2025.3621246 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-02. Verfasst von Weixin Ye und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: Fundamental Research Funds for the Central Universities, National NSF of China, Natural Science Foundation of Beijing, China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In federated learning, Transformer, as a popular architecture, faces critical challenges in defending against gradient attacks and improving model performance in both Computer Vision (CV) and Natural Language Processing (NLP) tasks. It has been revealed that the gradient of Position Embeddings (PEs) in Transformer contains sufficient information, which can be used to reconstruct the input data. To mitigate this issue, we introduce a Masked Jigsaw Puzzle (MJP) framework. MJP starts with random token shuffling to break the token order, and then a learnable unknown (unk) position embedding is used to mask out the PEs of the shuffled tokens. In this manner, the local spatial information which is encoded in the position embeddings is disrupted, and the models are forced to learn feature representations that are less reliant on the local spatial information. Notably, with the careful use of MJP, we can not only improve models' robustness against gradient attacks, but also boost their performance in both vision and text application scenarios, such as classification for images (e.g., ImageNet-1 K) and sentiment analysis for text (e.g., Yelp and Amazon). Experimental results suggest that MJP is a unified framework for different Transformer-based models in both vision and language tasks.

Förderung

  • Fundamental Research Funds for the Central Universities
  • National NSF of China
  • Natural Science Foundation of Beijing, China

Zitieren

APA

Ye, W., Wang, W., Liu, Y., Song, Y., Ren, B., Bi, W., Cucchiara, R., & Sebe, N. (2026). A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(2), 1873-1887. https://doi.org/10.1109/tpami.2025.3621246

Vancouver

Ye W, Wang W, Liu Y, Song Y, Ren B, Bi W, et al. A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(2):1873-1887. doi:10.1109/tpami.2025.3621246

BibTeX
@article{Ye2026,
  author = {Ye, Weixin and Wang, Wei and Liu, Yahui and Song, Yue and Ren, Bin and Bi, Wei and Cucchiara, Rita and Sebe, Nicu},
  title = {A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {2},
  pages = {1873--1887},
  doi = {10.1109/tpami.2025.3621246}
}
Kostenlos starten →