A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-02 · 48 (2) · 1873-1887
DOI: 10.1109/tpami.2025.3621246 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-02. Verfasst von Weixin Ye und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: Fundamental Research Funds for the Central Universities, National NSF of China, Natural Science Foundation of Beijing, China.
Abstract (Originalsprache)
In federated learning, Transformer, as a popular architecture, faces critical challenges in defending against gradient attacks and improving model performance in both Computer Vision (CV) and Natural Language Processing (NLP) tasks. It has been revealed that the gradient of Position Embeddings (PEs) in Transformer contains sufficient information, which can be used to reconstruct the input data. To mitigate this issue, we introduce a Masked Jigsaw Puzzle (MJP) framework. MJP starts with random token shuffling to break the token order, and then a learnable unknown (unk) position embedding is used to mask out the PEs of the shuffled tokens. In this manner, the local spatial information which is encoded in the position embeddings is disrupted, and the models are forced to learn feature representations that are less reliant on the local spatial information. Notably, with the careful use of MJP, we can not only improve models' robustness against gradient attacks, but also boost their performance in both vision and text application scenarios, such as classification for images (e.g., ImageNet-1 K) and sentiment analysis for text (e.g., Yelp and Amazon). Experimental results suggest that MJP is a unified framework for different Transformer-based models in both vision and language tasks.
Förderung
- Fundamental Research Funds for the Central Universities
- National NSF of China
- Natural Science Foundation of Beijing, China
Zitieren
APA
Ye, W., Wang, W., Liu, Y., Song, Y., Ren, B., Bi, W., Cucchiara, R., & Sebe, N. (2026). A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(2), 1873-1887. https://doi.org/10.1109/tpami.2025.3621246
Vancouver
Ye W, Wang W, Liu Y, Song Y, Ren B, Bi W, et al. A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(2):1873-1887. doi:10.1109/tpami.2025.3621246
BibTeX
@article{Ye2026,
author = {Ye, Weixin and Wang, Wei and Liu, Yahui and Song, Yue and Ren, Bin and Bi, Wei and Cucchiara, Rita and Sebe, Nicu},
title = {A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
volume = {48},
number = {2},
pages = {1873--1887},
doi = {10.1109/tpami.2025.3621246}
}
EUVIMED