EUVIMEDEUVIMED entdecken
Crossref

Batch Normalization Preconditioning for Stochastic Gradient Langevin Dynamics

Susanna Lange, Wei Deng, Qiang Ye, Guang Lin

Journal of Machine Learning · 2024

DOI: 10.4208/jml.220726a ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Machine Learning, 2024. Verfasst von Susanna Lange und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 25 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Stochastic gradient Langevin dynamics (SGLD) is a standard sampling technique for uncertainty estimation in Bayesian neural networks. Past methods have shown improved convergence by including a preconditioning of SGLD based on RMSprop. This preconditioning serves to adapt to the local geometry of the parameter space and improve the performance of deep neural networks. In this paper, we develop another preconditioning technique to accelerate training and improve convergence by incorporating a recently developed batch normalization preconditioning (BNP), into our methods. BNP uses mini-batch statistics to improve the conditioning of the Hessian of the loss function in traditional neural networks and thus improve convergence. We will show that applying BNP to SGLD will improve the conditioning of the Fisher information matrix, which improves the convergence. We present the results of this method on three experiments including a simulation example, a contextual bandit example, and a residual network which show the improved initial convergence provided by BNP, in addition to an improved condition number from this method.

Zitieren

APA

Lange, S., Deng, W., Ye, Q., & Lin, G. (2024). Batch Normalization Preconditioning for Stochastic Gradient Langevin Dynamics. Journal of Machine Learning. https://doi.org/10.4208/jml.220726a

Vancouver

Lange S, Deng W, Ye Q, Lin G. Batch Normalization Preconditioning for Stochastic Gradient Langevin Dynamics. Journal of Machine Learning. 2024. doi:10.4208/jml.220726a

BibTeX
@article{Lange2024,
  author = {Lange, Susanna and Deng, Wei and Ye, Qiang and Lin, Guang},
  title = {Batch Normalization Preconditioning for Stochastic Gradient Langevin Dynamics},
  journal = {Journal of Machine Learning},
  year = {2024},
  doi = {10.4208/jml.220726a}
}
Kostenlos starten →