EUVIMEDEUVIMED entdecken
Crossref

Fast-FNet: Accelerating transformer encoder models via efficient fourier layers

Nurullah Sevim, Ege Ozan Ozyedek, Furkan Şahinuç, Tuna Ozates, Aykut Koç

Signal, Image and Video Processing · 2025-12 · 19 (12)

DOI: 10.1007/s11760-025-04552-3 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Signal, Image and Video Processing, online erschienen am 03.09.2025. Verfasst von Nurullah Sevim und 4 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: A. Koc acknowledges support from the 2023 Young Scientist Award (BAGEP)., TUBITAK 1001 Grant.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformer-based language models utilize the attention mechanism for substantial performance improvements in almost all natural language processing (NLP) tasks. Similar attention structures are also extensively studied in several other areas. Although the attention mechanism enhances the model performances significantly, its quadratic complexity prevents efficient processing of long sequences. Recent works focused on eliminating the disadvantages of computational inefficiency and showed that transformer-based models can still reach competitive results without the attention layer. A pioneering study proposed the FNet, which replaces the attention layer with the Fourier Transform (FT) in the transformer encoder architecture. FNet achieves competitive performances concerning the original transformer encoder model while accelerating training process by removing the computational burden of the attention mechanism. However, the FNet model ignores essential properties of the FT from the classical signal processing that can be leveraged to increase model efficiency further. We propose different methods to deploy FT efficiently in transformer encoder models. Our proposed architectures have smaller number of model parameters, shorter training times, less memory usage, and some additional performance improvements. We demonstrate these improvements through extensive experiments on common benchmarks.

Förderung

  • A. Koc acknowledges support from the 2023 Young Scientist Award (BAGEP).
  • TUBITAK 1001 Grant

Zitieren

APA

Sevim, N., Ozyedek, E. O., Şahinuç, F., Ozates, T., & Koç, A. (2025). Fast-FNet: Accelerating transformer encoder models via efficient fourier layers. Signal, Image and Video Processing, 19(12). https://doi.org/10.1007/s11760-025-04552-3

Vancouver

Sevim N, Ozyedek EO, Şahinuç F, Ozates T, Koç A. Fast-FNet: Accelerating transformer encoder models via efficient fourier layers. Signal, Image and Video Processing. 2025;19(12). doi:10.1007/s11760-025-04552-3

BibTeX
@article{Sevim2025,
  author = {Sevim, Nurullah and Ozyedek, Ege Ozan and Şahinuç, Furkan and Ozates, Tuna and Koç, Aykut},
  title = {Fast-FNet: Accelerating transformer encoder models via efficient fourier layers},
  journal = {Signal, Image and Video Processing},
  year = {2025},
  volume = {19},
  number = {12},
  doi = {10.1007/s11760-025-04552-3}
}
Kostenlos starten →