Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models
International Journal of Scientific Research in Science and Technology · 19.04.2026 · 12 (5) · 864-871
DOI: 10.32628/ijsrst52310283 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Scientific Research in Science and Technology, online erschienen am 19.04.2026. Verfasst von Navya Veginati. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models based on transformers have shown outstanding performance in several natural language processing tasks. But, when such models are fine-tuned, catastrophic forgetting is likely to occur causing one to forget the knowledge that has been learned before. This paper suggests a better attention-based model to enhance the knowledge retention in fine-tuned transformer models. The method combines attention mechanisms that are aware of retention and knowledge distillation methods to retain important contextual information. The proposed model successfully balances knowledge preservation and new learning by proposing adaptive attention scaling and feature alignment strategies. Experimental evidence indicates that the models have highly improved accuracy, precision, recall, F1-score, and knowledge retention over existing models, including BERT, DistilBERT, and TinyBERT. The results indicate that the use of attention enhancement and learning through retention awareness is effective, offering an effective solution to knowledge decline in large language models.
Zitieren
APA
Veginati, N. (2026). Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models. International Journal of Scientific Research in Science and Technology, 12(5), 864-871. https://doi.org/10.32628/ijsrst52310283
Vancouver
Veginati N. Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models. International Journal of Scientific Research in Science and Technology. 2026;12(5):864-871. doi:10.32628/ijsrst52310283
BibTeX
@article{Veginati2026,
author = {Veginati, Navya},
title = {Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models},
journal = {International Journal of Scientific Research in Science and Technology},
year = {2026},
volume = {12},
number = {5},
pages = {864--871},
doi = {10.32628/ijsrst52310283}
}
EUVIMED