Survey of Optimization Algorithms in Modern Neural Networks
Mathematics · 26.05.2023 · 11 (11) · 2466
DOI: 10.3390/math11112466 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematics, online erschienen am 26.05.2023. Verfasst von Ruslan Abdulkadirov und 2 weiteren Autor:innen. Die Arbeit wurde bisher 112-mal zitiert (Crossref) und nennt 214 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Council for grants of President of Russian Federation, Russian Science Foundation.
Abstract (Originalsprache)
The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many problems, remains the most well-utilized branch of machine learning. Thus, one must select appropriate neural network architectures, data processing, and advanced applied mathematics tools. A common challenge for these networks is achieving the highest accuracy in a short time. This problem is solved by modifying networks and improving data pre-processing, where accuracy increases along with training time. Bt using optimization methods, one can improve the accuracy without increasing the time. In this review, we consider all existing optimization algorithms that meet in neural networks. We present modifications of optimization algorithms of the first, second, and information-geometric order, which are related to information geometry for Fisher–Rao and Bregman metrics. These optimizers have significantly influenced the development of neural networks through geometric and probabilistic tools. We present applications of all the given optimization algorithms, considering the types of neural networks. After that, we show ways to develop optimization algorithms in further research using modern neural networks. Fractional order, bilevel, and gradient-free optimizers can replace classical gradient-based optimizers. Such approaches are induced in graph, spiking, complex-valued, quantum, and wavelet neural networks. Besides pattern recognition, time series prediction, and object detection, there are many other applications in machine learning: quantum computations, partial differential, and integrodifferential equations, and stochastic processes.
Förderung
- Council for grants of President of Russian Federation
- Russian Science Foundation
Zitieren
APA
Abdulkadirov, R., Lyakhov, P., & Nagornov, N. (2023). Survey of Optimization Algorithms in Modern Neural Networks. Mathematics, 11(11), 2466. https://doi.org/10.3390/math11112466
Vancouver
Abdulkadirov R, Lyakhov P, Nagornov N. Survey of Optimization Algorithms in Modern Neural Networks. Mathematics. 2023;11(11):2466. doi:10.3390/math11112466
BibTeX
@article{Abdulkadirov2023,
author = {Abdulkadirov, Ruslan and Lyakhov, Pavel and Nagornov, Nikolay},
title = {Survey of Optimization Algorithms in Modern Neural Networks},
journal = {Mathematics},
year = {2023},
volume = {11},
number = {11},
pages = {2466},
doi = {10.3390/math11112466}
}
EUVIMED