Comparative Analysis of First and Second Order Methods for Optimization in Neural Networks
Proceedings of International Mathematical Sciences · 2022
DOI: 10.47086/pims.1170457 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of International Mathematical Sciences, 2022. Verfasst von Auras KHANAL und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 13 Literaturangaben. Gefördert von: Beloit College.
Abstract (Originalsprache)
Artificial Neural Networks are fine tuned to yield the best performance through an iterative process where the values of their parameters are altered. Optimization is the preferred method to determine the parameters that yield the minima of the loss function, an evaluation metric for ANN’s. However, the process of finding an optimal model which has minimum loss faces several obstacles, the most notable being the efficiency and rate of convergence to the minima of the loss function. Such optimization efficiency is imperative to reduce the use of computational resources and time when training Neural Network models. This paper reviews and compares the intuition and effectiveness of existing optimization algorithms such as Gradient Descent, Gradient Descent with Momentum, RMSProp and Adam that implement first order derivatives, and Newton’s Method that utilizes second order derivatives for convergence. It also explores the possibility to combine and leverage first and second order optimization techniques for improved performance when training Artificial Neural Networks.
Förderung
- Beloit College
Zitieren
APA
KHANAL, A., & DİK, M. (2022). Comparative Analysis of First and Second Order Methods for Optimization in Neural Networks. Proceedings of International Mathematical Sciences. https://doi.org/10.47086/pims.1170457
Vancouver
KHANAL A, DİK M. Comparative Analysis of First and Second Order Methods for Optimization in Neural Networks. Proceedings of International Mathematical Sciences. 2022. doi:10.47086/pims.1170457
BibTeX
@article{KHANAL2022,
author = {KHANAL, Auras and DİK, Mehmet},
title = {Comparative Analysis of First and Second Order Methods for Optimization in Neural Networks},
journal = {Proceedings of International Mathematical Sciences},
year = {2022},
doi = {10.47086/pims.1170457}
}
EUVIMED