EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches

Zhaoyan Lyu iD, Gholamali Aminian iD, Miguel R D Rodrigues iD

Entropy · 14.07.2023 · 25 (7) · 1063

DOI: 10.3390/e25071063 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Entropy, online erschienen am 14.07.2023. Verfasst von Zhaoyan Lyu und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electronic and Electrical Engineering, University College London, Gower St., London WC1E 6BT, UK. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: UCL Dean’s Prize, China Scholarship Council.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

It is well-known that a neural network learning process—along with its connections to fitting, compression, and generalization—is not yet well understood. In this paper, we propose a novel approach to capturing such neural network dynamics using information-bottleneck-type techniques, involving the replacement of mutual information measures (which are notoriously difficult to estimate in high-dimensional spaces) by other more tractable ones, including (1) the minimum mean-squared error associated with the reconstruction of the network input data from some intermediate network representation and (2) the cross-entropy associated with a certain class label given some network representation. We then conducted an empirical study in order to ascertain how different network models, network learning algorithms, and datasets may affect the learning dynamics. Our experiments show that our proposed approach appears to be more reliable in comparison with classical information bottleneck ones in capturing network dynamics during both the training and testing phases. Our experiments also reveal that the fitting and compression phases exist regardless of the choice of activation function. Additionally, our findings suggest that model architectures, training algorithms, and datasets that lead to better generalization tend to exhibit more pronounced fitting and compression phases.

Schlagwörter

Stichwörter der Autor:innen

compressiondeep learningfittinggeneralizationinformation bottleneckinformation theory

Autor:innen und Institutionen

  • Department of Electronic and Electrical Engineering, University College London, Gower St., London WC1E 6BT, UK.
  • The Alan Turing Institute, British Library, 96 Euston Rd., London NW1 2DB, UK.

ORCID-Profile: Zhaoyan Lyu, Gholamali Aminian, Miguel R D Rodrigues

Förderung

  • UCL Dean’s Prize
  • China Scholarship Council

Zitieren

APA

Lyu, Z., Aminian, G., & Rodrigues, M. R. D. (2023). On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches. Entropy, 25(7), 1063. https://doi.org/10.3390/e25071063

Vancouver

Lyu Z, Aminian G, Rodrigues MRD. On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches. Entropy. 2023;25(7):1063. doi:10.3390/e25071063

BibTeX
@article{Lyu2023,
  author = {Lyu, Zhaoyan and Aminian, Gholamali and Rodrigues, Miguel R D},
  title = {On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches},
  journal = {Entropy},
  year = {2023},
  volume = {25},
  number = {7},
  pages = {1063},
  doi = {10.3390/e25071063}
}
Kostenlos starten →