EUVIMEDEUVIMED entdecken
Crossref

On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD

Tongcheng Zhang, Zhanpeng Zhou, Mingze Wang, Andi Han, Wei Huang, Taiji Suzuki, Junchi Yan

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (33) · 28400-28408

DOI: 10.1609/aaai.v40i33.40069 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Tongcheng Zhang und 6 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

One crucial factor behind the success of deep learning lies in the implicit bias induced by noise inherent in gradient-based training algorithms. Motivated by empirical observations that training with noisy labels improves model generalization, we delve into the underlying mechanisms behind stochastic gradient descent (SGD) with label noise. Focusing on a two-layer over-parameterized linear network, we analyze the learning dynamics of label noise SGD, unveiling a two-phase learning behavior. In Phase I, the magnitudes of model weights progressively diminish, and the model escapes the lazy regime; enters the rich regime. In Phase II, the alignment between model weights and the ground-truth interpolator increases, and the model eventually converges. Our analysis highlights the critical role of label noise in driving the transition from the lazy to the rich regime and minimally explains its empirical success. Furthermore, we extend these insights to Sharpness-Aware Minimization (SAM), showing that the principles governing label noise SGD also apply to broader optimization algorithms. Extensive experiments, conducted under both synthetic and real-world setups, strongly support our theory.

Zitieren

APA

Zhang, T., Zhou, Z., Wang, M., Han, A., Huang, W., Suzuki, T., & Yan, J. (2026). On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD. Proceedings of the AAAI Conference on Artificial Intelligence, 40(33), 28400-28408. https://doi.org/10.1609/aaai.v40i33.40069

Vancouver

Zhang T, Zhou Z, Wang M, Han A, Huang W, Suzuki T, et al. On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(33):28400-28408. doi:10.1609/aaai.v40i33.40069

BibTeX
@article{Zhang2026,
  author = {Zhang, Tongcheng and Zhou, Zhanpeng and Wang, Mingze and Han, Andi and Huang, Wei and Suzuki, Taiji and Yan, Junchi},
  title = {On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {33},
  pages = {28400--28408},
  doi = {10.1609/aaai.v40i33.40069}
}
Kostenlos starten →