EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Cognitive Load Theory-Informed Attention Mechanism for Transformer-Based Text Classification

Jarrod Graham, Victor S. Sheng

Mathematics · 28.03.2026 · 14 (7) · 1133

DOI: 10.3390/math14071133 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mathematics, online erschienen am 28.03.2026. Verfasst von Jarrod Graham und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We propose a Cognitive Load Theory (CLT)-informed attention mechanism for transformer-based text classification. The proposed attention mechanism computes a per-token cognitive-load signal—derived from attention entropy, margin-based classification uncertainty, and optional inverse document frequency—and maps this signal to a learnable attention “budget” that scales outgoing attention mass during decoding. Unlike architectural efficiency techniques such as Multi-Query or Grouped-Query Attention, the CLT mechanism requires no structural modifications and introduces only modest per-step computational overhead while preserving full compatibility with standard transformer architectures. Experiments across four datasets (IMDB, AG News, SST-2, and DBpedia) show that CLT-informed attention achieves accuracy comparable to or exceeding a fixed-budget baseline while delivering consistently lower test loss, faster convergence to the best validation checkpoint, reduced attention entropy, and strong alignment between cognitive load and attention mass. Among all variants, an entropy-only load signal yields the most stable and consistent performance across datasets. These results demonstrate that lightweight, cognitively motivated constraints can structure transformer attention while maintaining or improving downstream classification performance.

Zitieren

APA

Graham, J., & Sheng, V. S. (2026). A Cognitive Load Theory-Informed Attention Mechanism for Transformer-Based Text Classification. Mathematics, 14(7), 1133. https://doi.org/10.3390/math14071133

Vancouver

Graham J, Sheng VS. A Cognitive Load Theory-Informed Attention Mechanism for Transformer-Based Text Classification. Mathematics. 2026;14(7):1133. doi:10.3390/math14071133

BibTeX
@article{Graham2026,
  author = {Graham, Jarrod and Sheng, Victor S.},
  title = {A Cognitive Load Theory-Informed Attention Mechanism for Transformer-Based Text Classification},
  journal = {Mathematics},
  year = {2026},
  volume = {14},
  number = {7},
  pages = {1133},
  doi = {10.3390/math14071133}
}
Kostenlos starten →