On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches
Entropy · 14.07.2023 · 25 (7) · 1063
DOI: 10.3390/e25071063 ↗Kurz gefasst
Fachartikel in der Zeitschrift Entropy, online erschienen am 14.07.2023. Verfasst von Zhaoyan Lyu und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electronic and Electrical Engineering, University College London, Gower St., London WC1E 6BT, UK. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: UCL Dean’s Prize, China Scholarship Council.
Abstract (Originalsprache)
It is well-known that a neural network learning process—along with its connections to fitting, compression, and generalization—is not yet well understood. In this paper, we propose a novel approach to capturing such neural network dynamics using information-bottleneck-type techniques, involving the replacement of mutual information measures (which are notoriously difficult to estimate in high-dimensional spaces) by other more tractable ones, including (1) the minimum mean-squared error associated with the reconstruction of the network input data from some intermediate network representation and (2) the cross-entropy associated with a certain class label given some network representation. We then conducted an empirical study in order to ascertain how different network models, network learning algorithms, and datasets may affect the learning dynamics. Our experiments show that our proposed approach appears to be more reliable in comparison with classical information bottleneck ones in capturing network dynamics during both the training and testing phases. Our experiments also reveal that the fitting and compression phases exist regardless of the choice of activation function. Additionally, our findings suggest that model architectures, training algorithms, and datasets that lead to better generalization tend to exhibit more pronounced fitting and compression phases.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Electronic and Electrical Engineering, University College London, Gower St., London WC1E 6BT, UK.
- The Alan Turing Institute, British Library, 96 Euston Rd., London NW1 2DB, UK.
ORCID-Profile: Zhaoyan Lyu, Gholamali Aminian, Miguel R D Rodrigues
Förderung
- UCL Dean’s Prize
- China Scholarship Council
Zitieren
APA
Lyu, Z., Aminian, G., & Rodrigues, M. R. D. (2023). On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches. Entropy, 25(7), 1063. https://doi.org/10.3390/e25071063
Vancouver
Lyu Z, Aminian G, Rodrigues MRD. On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches. Entropy. 2023;25(7):1063. doi:10.3390/e25071063
BibTeX
@article{Lyu2023,
author = {Lyu, Zhaoyan and Aminian, Gholamali and Rodrigues, Miguel R D},
title = {On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches},
journal = {Entropy},
year = {2023},
volume = {25},
number = {7},
pages = {1063},
doi = {10.3390/e25071063}
}
EUVIMED