Learning Rate Curriculum
International Journal of Computer Vision · 2025-01 · 133 (1) · 291-314
DOI: 10.1007/s11263-024-02186-5 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 27.07.2024. Verfasst von Florinel-Alin Croitoru und 3 weiteren Autor:innen. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 72 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: UEFISCDI.
Abstract (Originalsprache)
Most curriculum learning methods require an approach to sort the data samples by difficulty, which is often cumbersome to perform. In this work, we propose a novel curriculum learning approach termed Learning Rate Curriculum (LeRaC), which leverages the use of a different learning rate for each layer of a neural network to create a data-agnostic curriculum during the initial training epochs. More specifically, LeRaC assigns higher learning rates to neural layers closer to the input, gradually decreasing the learning rates as the layers are placed farther away from the input. The learning rates increase at various paces during the first training iterations, until they all reach the same value. From this point on, the neural model is trained as usual. This creates a model-level curriculum learning strategy that does not require sorting the examples by difficulty and is compatible with any neural network, generating higher performance levels regardless of the architecture. We conduct comprehensive experiments on 12 data sets from the computer vision (CIFAR-10, CIFAR-100, Tiny ImageNet, ImageNet-1K, Food-101, UTKFace, PASCAL VOC), language (BoolQ, QNLI, RTE) and audio (ESC-50, CREMA-D) domains, considering various convolutional (ResNet-18, Wide-ResNet-50, DenseNet-121, YOLOv5), recurrent (LSTM) and transformer (CvT, BERT, SepTr) architectures. We compare our approach with the conventional training regime, as well as with Curriculum by Smoothing (CBS), a state-of-the-art data-agnostic curriculum learning approach. Unlike CBS, our performance improvements over the standard training regime are consistent across all data sets and models. Furthermore, we significantly surpass CBS in terms of training time (there is no additional cost over the standard training regime for LeRaC). Our code is freely available at: https://github.com/CroitoruAlin/LeRaC .
Förderung
- UEFISCDI
Zitieren
APA
Croitoru, F., Ristea, N., Ionescu, R. T., & Sebe, N. (2025). Learning Rate Curriculum. International Journal of Computer Vision, 133(1), 291-314. https://doi.org/10.1007/s11263-024-02186-5
Vancouver
Croitoru F, Ristea N, Ionescu RT, Sebe N. Learning Rate Curriculum. International Journal of Computer Vision. 2025;133(1):291-314. doi:10.1007/s11263-024-02186-5
BibTeX
@article{Croitoru2025,
author = {Croitoru, Florinel-Alin and Ristea, Nicolae-Cătălin and Ionescu, Radu Tudor and Sebe, Nicu},
title = {Learning Rate Curriculum},
journal = {International Journal of Computer Vision},
year = {2025},
volume = {133},
number = {1},
pages = {291--314},
doi = {10.1007/s11263-024-02186-5}
}
EUVIMED