Token-Context Attention for NLI: An Alternative to Self-Attention
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (41) · 34836-34844
DOI: 10.1609/aaai.v40i41.40786 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xin Zhang und 1 weiteren Autor:innen.
Abstract (Originalsprache)
Despite the rapid progress in large language models (LLMs), even sub-billion-scale systems perform at chance level on challenging natural language inference (NLI) benchmarks such as Adversarial Natural Language Inference (ANLI), while training larger models is often impractical due to limited computational resources. We address this parameter-efficiency bottleneck in NLI with a Complex-Vector Token Representation that explicitly decouples each token from its context, and a Token-Context Attention mechanism that updates each token based on the most informative contextual semantics. On ANLI, a 0.8B-parameter Token-Context Attention model achieves higher parameter efficiency (accuracy per parameter) than all 1B and comparable 0.8B self-attention baselines; it also suffers smaller performance degradation under Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks and achieves the largest few-shot gains on SNLI and MNLI while exhibiting no significant degradation in ANLI accuracy after adaptation. These results suggest that explicitly disentangling token and context offers a viable alternative to standard self-attention for NLI tasks.
Zitieren
APA
Zhang, X., & Sheng, V. S. (2026). Token-Context Attention for NLI: An Alternative to Self-Attention. Proceedings of the AAAI Conference on Artificial Intelligence, 40(41), 34836-34844. https://doi.org/10.1609/aaai.v40i41.40786
Vancouver
Zhang X, Sheng VS. Token-Context Attention for NLI: An Alternative to Self-Attention. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(41):34836-34844. doi:10.1609/aaai.v40i41.40786
BibTeX
@article{Zhang2026,
author = {Zhang, Xin and Sheng, Victor S.},
title = {Token-Context Attention for NLI: An Alternative to Self-Attention},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {41},
pages = {34836--34844},
doi = {10.1609/aaai.v40i41.40786}
}
EUVIMED