Chrono Initialized LSTM Networks With Layer Normalization
IEEE Access · 2024 · 12 · 115219-115236
DOI: 10.1109/access.2024.3445329 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2024. Verfasst von Antonio Tolic und 2 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Recurrent Neural Networks (RNNs), including the distinguished Long Short-Term Memory Networks (LSTMs), have been shown to be effective in a wide range of sequential data problems. However, their ability to model very long-term dependencies still poses challenges, indicating that this remains an active area of research. We propose a new methodology for gradient propagation in LSTM networks that addresses the limitations of traditional approaches. This methodology employs Chrono Initialization (CI) and Layer Normalization (LN) techniques for LSTM networks. CI ensures that the gradients are neither too small nor too large, preventing gradient vanishing or exploding, while LN enhances the stability of hidden state dynamics by aligning the data distribution, mitigating gradient-related issues, and facilitating faster learning. The proposed approach consistently outperformed baseline models in our comparative analyses, achieving average accuracy improvements across classification tasks, with improvements of up to 5% and notable performance increases exceeding 30% in certain scenarios. Significant reductions in Mean Squared Error (MSE) were persistently achieved across regression tasks, averaging 35% lower to several orders of magnitude lower MSE, especially when the baseline models underperformed. Moreover, the method significantly improved performance in sequence generation tasks, consistently yielding much lower negative log-likelihoods relative to the reference counterparts, often by several orders of magnitude. Faster convergence to the optimal minimum was also repeatedly confirmed, even when final results were not significantly divergent. Overall, this new methodology improves the performance of LSTM networks and has been evaluated across various sequential learning tasks. A formal analysis provides a deeper understanding of the mechanisms behind these improvements.
Zitieren
APA
Tolic, A., Boshkoska, B. M., & Skansi, S. (2024). Chrono Initialized LSTM Networks With Layer Normalization. IEEE Access, 12, 115219-115236. https://doi.org/10.1109/access.2024.3445329
Vancouver
Tolic A, Boshkoska BM, Skansi S. Chrono Initialized LSTM Networks With Layer Normalization. IEEE Access. 2024;12:115219-115236. doi:10.1109/access.2024.3445329
BibTeX
@article{Tolic2024,
author = {Tolic, Antonio and Boshkoska, Biljana Mileva and Skansi, Sandro},
title = {Chrono Initialized LSTM Networks With Layer Normalization},
journal = {IEEE Access},
year = {2024},
volume = {12},
pages = {115219--115236},
doi = {10.1109/access.2024.3445329}
}
EUVIMED