EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Explaining neural scaling laws

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee iD, Utkarsh Sharma

Proceedings of the National Academy of Sciences · 02.07.2024 · 121 (27)

DOI: 10.1073/pnas.2311878121 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, online erschienen am 24.06.2024. Verfasst von Yasaman Bahri und 4 weiteren Autor:innen. Beteiligte Einrichtung: Google DeepMind, Mountain View, CA 94043. Die Arbeit wurde bisher 153-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents under modifications of task and architecture aspect ratio. Our work provides a taxonomy for classifying different scaling regimes, underscores that there can be different mechanisms driving improvements in loss, and lends insight into the microscopic origin and relationships between scaling exponents.

Schlagwörter

Stichwörter der Autor:innen

deep neural networksmachine learningstatistical physics

Autor:innen und Institutionen

  • Google DeepMind, Mountain View, CA 94043.
  • Department of Physics and Astronomy, Johns Hopkins University, Baltimore, MD 21218.

ORCID-Profile: Jaehoon Lee

Zitieren

APA

Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2024). Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27). https://doi.org/10.1073/pnas.2311878121

Vancouver

Bahri Y, Dyer E, Kaplan J, Lee J, Sharma U. Explaining neural scaling laws. Proceedings of the National Academy of Sciences. 2024;121(27). doi:10.1073/pnas.2311878121

BibTeX
@article{Bahri2024,
  author = {Bahri, Yasaman and Dyer, Ethan and Kaplan, Jared and Lee, Jaehoon and Sharma, Utkarsh},
  title = {Explaining neural scaling laws},
  journal = {Proceedings of the National Academy of Sciences},
  year = {2024},
  volume = {121},
  number = {27},
  doi = {10.1073/pnas.2311878121}
}
Kostenlos starten →