On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (33) · 28400-28408
DOI: 10.1609/aaai.v40i33.40069 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Tongcheng Zhang und 6 weiteren Autor:innen.
Abstract (Originalsprache)
One crucial factor behind the success of deep learning lies in the implicit bias induced by noise inherent in gradient-based training algorithms. Motivated by empirical observations that training with noisy labels improves model generalization, we delve into the underlying mechanisms behind stochastic gradient descent (SGD) with label noise. Focusing on a two-layer over-parameterized linear network, we analyze the learning dynamics of label noise SGD, unveiling a two-phase learning behavior. In Phase I, the magnitudes of model weights progressively diminish, and the model escapes the lazy regime; enters the rich regime. In Phase II, the alignment between model weights and the ground-truth interpolator increases, and the model eventually converges. Our analysis highlights the critical role of label noise in driving the transition from the lazy to the rich regime and minimally explains its empirical success. Furthermore, we extend these insights to Sharpness-Aware Minimization (SAM), showing that the principles governing label noise SGD also apply to broader optimization algorithms. Extensive experiments, conducted under both synthetic and real-world setups, strongly support our theory.
Zitieren
APA
Zhang, T., Zhou, Z., Wang, M., Han, A., Huang, W., Suzuki, T., & Yan, J. (2026). On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD. Proceedings of the AAAI Conference on Artificial Intelligence, 40(33), 28400-28408. https://doi.org/10.1609/aaai.v40i33.40069
Vancouver
Zhang T, Zhou Z, Wang M, Han A, Huang W, Suzuki T, et al. On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(33):28400-28408. doi:10.1609/aaai.v40i33.40069
BibTeX
@article{Zhang2026,
author = {Zhang, Tongcheng and Zhou, Zhanpeng and Wang, Mingze and Han, Andi and Huang, Wei and Suzuki, Taiji and Yan, Junchi},
title = {On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {33},
pages = {28400--28408},
doi = {10.1609/aaai.v40i33.40069}
}
EUVIMED