EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Orthogonal position representations for transformer in neural machine translation

Yue Zhao iD, Qinghong Zhang, Shuhan Zhou

PLOS One · 2025

DOI: 10.1371/journal.pone.0334443 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Yue Zhao und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Marxism, Northeastern University, Shenyang, China. Hauptthemen laut MEDLINE: Neural Networks, Computer. Die Arbeit nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, the Transformer architecture has solidified its position as the dominant model in neural machine translation (NMT), thanks to its exceptional effectiveness in capturing long-range dependencies and remarkable scalability across diverse linguistic tasks. A key characteristic of the Transformer is its reliance on self-attention mechanisms, which, while powerful, are inherently position-insensitive—treating tokens as a set rather than an ordered sequence. This limitation makes positional encoding a critical component in Transformer-based models and their variants, as it provides the necessary sequential context to differentiate token positions within a sequence. In this paper, we address this challenge by proposing a novel orthogonal fixed-dimension positional representation (OPR). This design is meticulously engineered to maximize the discrimination of positions within a sequence, ensuring that each position is uniquely and distinctively encoded. Notably, OPR introduces no additional parameters to the model and incurs no extra computational overhead, making it highly efficient for real-world applications. Our experimental evaluations, conducted across multiple standard NMT datasets, demonstrate that OPR consistently outperforms several strong baselines, including traditional sine-cosine positional encoding and learnable positional embeddings. It achieves notable improvements in both BLEU and COMET scores, with gains observed across all tested language pairs. Furthermore, when combined with relative positional encoding (RPR), the OPR method’s performance is further enhanced, highlighting its ability to effectively model both absolute and relative positional relationships—a dual capability that is crucial for nuanced sequence understanding.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Neural Networks, ComputerHumansAlgorithmsLinguisticsLanguage

Autor:innen und Institutionen

  • School of Marxism, Northeastern University, Shenyang, China.
  • School of Computer Science and Engineering, Northeastern University, Shenyang, China.

ORCID-Profile: Yue Zhao

Zitieren

APA

Zhao, Y., Zhang, Q., & Zhou, S. (2025). Orthogonal position representations for transformer in neural machine translation. PLOS One. https://doi.org/10.1371/journal.pone.0334443

Vancouver

Zhao Y, Zhang Q, Zhou S. Orthogonal position representations for transformer in neural machine translation. PLOS One. 2025. doi:10.1371/journal.pone.0334443

BibTeX
@article{Zhao2025,
  author = {Zhao, Yue and Zhang, Qinghong and Zhou, Shuhan},
  title = {Orthogonal position representations for transformer in neural machine translation},
  journal = {PLOS One},
  year = {2025},
  doi = {10.1371/journal.pone.0334443}
}
Kostenlos starten →