Assessing the role of key and value dimensionality in multi-head attention mechanisms
Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi · 14.07.2026 · (Advanced Online Publication)
DOI: 10.65206/pajes.1920115 ↗Kurz gefasst
Fachartikel in der Zeitschrift Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi, online erschienen am 14.07.2026. Verfasst von Ramazan Katırcı und 2 weiteren Autor:innen. Die Arbeit nennt 25 Literaturangaben. Gefördert von: This work has been supported by the Scientific Research Projects Coordination Unit of the Sivas University of Science and Technology..
Abstract (Originalsprache)
Context—The transformer architecture has occupied a prominent position in recent research due to its attention-driven design; however, the role of asymmetric key and value dimensions within its attention mechanism remains largely underexplored in the literature despite their potential impact on model behavior and performance.Objective—This study aims to systematically analyze the effects of key architectural parameters in the Transformer architecture on neural machine translation performance. These parameters include asymmetric key and value dimensions within the multi-head attention mechanism, embedding dimension, number of attention heads and sentence length.Method—This study analyses the architectural components of the Transformer model across three neural machine translation datasets: English–Turkish, French–Turkish, and German–Turkish. Six fundamental hyperparameters—source and target embedding dimensions, key and value dimensions, the number of attention heads, and sentence length—were assessed at two levels each. To address the computational complexity of the Transformer's modular structure, a fractional factorial experimental design was employed, reducing the 64 possible configurations to 16 experimental runs. This approach significantly reduced the use of computational resources while ensuring a comprehensive analysis of both individual parameters and their pairwise interactions. All models were trained for 150 epochs using PyTorch with Distributed Data Parallel (DDP) and Automated Mixed Precision (AMP) in an HPC environment with NVIDIA A100 GPUs. Finally, statistical evaluation and performance visualization were conducted using Minitab.Results—The results show that while establishing equivalent key and value dimensions serves as a stable default, it does not guarantee optimal performance across linguistic contexts. The model’s response to asymmetrical configurations was highly language-specific, with notable variability observed in the interaction between the key dimension and the number of attention heads. Furthermore, increasing the target embedding dimension consistently improved generalization, whereas a larger source embedding dimension led to overfitting across all language pairs. These effects resulted from specific combinations of parameters that improved training accuracy but reduced validation performance, reflecting limitations in the generalization of language-dependent processes.Conclusion—These results emphasize that optimal Transformer performance cannot be achieved through a universal hyperparameter configuration and must instead be tailored to the linguistic characteristics of each dataset.
Förderung
- This work has been supported by the Scientific Research Projects Coordination Unit of the Sivas University of Science and Technology.
Zitieren
APA
Katırcı, R., Çelik, H., & Gön, N. (2026). Assessing the role of key and value dimensionality in multi-head attention mechanisms. Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi. https://doi.org/10.65206/pajes.1920115
Vancouver
Katırcı R, Çelik H, Gön N. Assessing the role of key and value dimensionality in multi-head attention mechanisms. Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi. 2026. doi:10.65206/pajes.1920115
BibTeX
@article{Katrc2026,
author = {Katırcı, Ramazan and Çelik, Hilal and Gön, Nusret},
title = {Assessing the role of key and value dimensionality in multi-head attention mechanisms},
journal = {Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi},
year = {2026},
number = {Advanced Online Publication},
doi = {10.65206/pajes.1920115}
}
EUVIMED