Rethinking Position Embedding Methods in the Transformer Architecture
Neural Processing Letters · 16.02.2024 · 56 (2)
DOI: 10.1007/s11063-024-11539-7 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neural Processing Letters, online erschienen am 16.02.2024. Verfasst von Xin Zhou und 5 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Science Foundation of China, Natural Science Foundation of Liaoning Province, Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
In the transformer architecture, as self-attention reads entire image patches at once, the context of the sequence between patches is omitted. Therefore, the position embedding method is employed to assist the self-attention layers in computing the ordering information of tokens. While many papers simply add the position vector to the corresponding token vector rather than concatenating them, few papers offer a thorough explanation and comparison beyond dimension reduction. However, the addition method is not meaningful because token vectors and position vectors are different physical quantities that cannot be directly combined through addition. Hence, we investigate the disparity in learnable absolute position information between the two embedding methods (concatenation and addition) and compare their performance on models. Experiments demonstrate that the concatenation method can learn more spatial information (such as horizontal, vertical, and angle) than the addition method. Furthermore, it reduces the attention distance in the final few layers. Moreover, the concatenation method exhibits greater robustness and leads to a performance gain of 0.1–0.5% for existing models without additional computation overhead.
Förderung
- Natural Science Foundation of China
- Natural Science Foundation of Liaoning Province
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Zhou, X., Ren, Z., Zhou, S., Jiang, Z., Yu, T., & Luo, H. (2024). Rethinking Position Embedding Methods in the Transformer Architecture. Neural Processing Letters, 56(2). https://doi.org/10.1007/s11063-024-11539-7
Vancouver
Zhou X, Ren Z, Zhou S, Jiang Z, Yu T, Luo H. Rethinking Position Embedding Methods in the Transformer Architecture. Neural Processing Letters. 2024;56(2). doi:10.1007/s11063-024-11539-7
BibTeX
@article{Zhou2024,
author = {Zhou, Xin and Ren, Zhaohui and Zhou, Shihua and Jiang, Zeyu and Yu, TianZhuang and Luo, Hengfa},
title = {Rethinking Position Embedding Methods in the Transformer Architecture},
journal = {Neural Processing Letters},
year = {2024},
volume = {56},
number = {2},
doi = {10.1007/s11063-024-11539-7}
}
EUVIMED