EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models

Navya Veginati

International Journal of Scientific Research in Science and Technology · 19.04.2026 · 12 (5) · 864-871

DOI: 10.32628/ijsrst52310283 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Scientific Research in Science and Technology, online erschienen am 19.04.2026. Verfasst von Navya Veginati. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models based on transformers have shown outstanding performance in several natural language processing tasks. But, when such models are fine-tuned, catastrophic forgetting is likely to occur causing one to forget the knowledge that has been learned before. This paper suggests a better attention-based model to enhance the knowledge retention in fine-tuned transformer models. The method combines attention mechanisms that are aware of retention and knowledge distillation methods to retain important contextual information. The proposed model successfully balances knowledge preservation and new learning by proposing adaptive attention scaling and feature alignment strategies. Experimental evidence indicates that the models have highly improved accuracy, precision, recall, F1-score, and knowledge retention over existing models, including BERT, DistilBERT, and TinyBERT. The results indicate that the use of attention enhancement and learning through retention awareness is effective, offering an effective solution to knowledge decline in large language models.

Zitieren

APA

Veginati, N. (2026). Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models. International Journal of Scientific Research in Science and Technology, 12(5), 864-871. https://doi.org/10.32628/ijsrst52310283

Vancouver

Veginati N. Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models. International Journal of Scientific Research in Science and Technology. 2026;12(5):864-871. doi:10.32628/ijsrst52310283

BibTeX
@article{Veginati2026,
  author = {Veginati, Navya},
  title = {Enhancing Transformer Attention Mechanisms for Knowledge Retention in Fine-Tuned Large Language Models},
  journal = {International Journal of Scientific Research in Science and Technology},
  year = {2026},
  volume = {12},
  number = {5},
  pages = {864--871},
  doi = {10.32628/ijsrst52310283}
}
Kostenlos starten →