EUVIMEDEUVIMED entdecken
Crossref

Token-Context Attention for NLI: An Alternative to Self-Attention

Xin Zhang, Victor S. Sheng

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (41) · 34836-34844

DOI: 10.1609/aaai.v40i41.40786 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xin Zhang und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Despite the rapid progress in large language models (LLMs), even sub-billion-scale systems perform at chance level on challenging natural language inference (NLI) benchmarks such as Adversarial Natural Language Inference (ANLI), while training larger models is often impractical due to limited computational resources. We address this parameter-efficiency bottleneck in NLI with a Complex-Vector Token Representation that explicitly decouples each token from its context, and a Token-Context Attention mechanism that updates each token based on the most informative contextual semantics. On ANLI, a 0.8B-parameter Token-Context Attention model achieves higher parameter efficiency (accuracy per parameter) than all 1B and comparable 0.8B self-attention baselines; it also suffers smaller performance degradation under Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks and achieves the largest few-shot gains on SNLI and MNLI while exhibiting no significant degradation in ANLI accuracy after adaptation. These results suggest that explicitly disentangling token and context offers a viable alternative to standard self-attention for NLI tasks.

Zitieren

APA

Zhang, X., & Sheng, V. S. (2026). Token-Context Attention for NLI: An Alternative to Self-Attention. Proceedings of the AAAI Conference on Artificial Intelligence, 40(41), 34836-34844. https://doi.org/10.1609/aaai.v40i41.40786

Vancouver

Zhang X, Sheng VS. Token-Context Attention for NLI: An Alternative to Self-Attention. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(41):34836-34844. doi:10.1609/aaai.v40i41.40786

BibTeX
@article{Zhang2026,
  author = {Zhang, Xin and Sheng, Victor S.},
  title = {Token-Context Attention for NLI: An Alternative to Self-Attention},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {41},
  pages = {34836--34844},
  doi = {10.1609/aaai.v40i41.40786}
}
Kostenlos starten →