EUVIMEDEUVIMED entdecken
Crossref

MRACL: Multi-Reward Space Guided Adaptive Curriculum Reinforcement Learning for LLMs

Wenxuan Liu, Liangyu Huo, Yi Jing, Xiyuan Zhang, Jian Xie

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (44) · 37663-37672

DOI: 10.1609/aaai.v40i44.41101 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Wenxuan Liu und 4 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Reinforcement learning (RL) has recently become a powerful yet resource-intensive approach for post-training large language models (LLMs). Incorporating curriculum learning (CL) into RL has been shown to significantly improve training efficiency, particularly in reasoning tasks. However, existing CL methods face substantial challenges in multi-objective RL (MORL) settings, including: (1) difficulty in evaluating model capabilities online, (2) challenges in assessing sample importance under diverse objectives, and (3) inherent trade-offs between online training and offline inference in dynamically designing the curriculum. To address these issues, we propose a Multi-Reward space guided Adaptive Curriculum Learning framework (MRACL), which is the first to incorporate curriculum learning into multi-objective RL. MRACL first constructs a multi-dimensional reward space via offline inference to establish initial reward profiles for each training sample. During training, based on reward space, it estimates the evolving model capabilities by computing the centroid of the space and calculates the sample priority score through its capability distance, optimization direction, and historical evolution, which enables adaptive selection of the most informative training samples at each step, independent of the specific RL algorithm. After each RL training iteration, the reward space is dynamically updated to reflect the model's evolving capabilities and the shifting distribution of sample priorities. Experiments on multi-objective alignment tasks demonstrate that MRACL achieves 1.62× faster convergence compared to state-of-the-art curriculum methods and 2.55× faster than non-curriculum methods. Furthermore, it consistently outperforms all baselines in both win rate and rule-based evaluation. We further provide an in-depth analysis of the key factors contributing to \modelname's effectiveness, along with its advantages, scenarios, and generalization across diverse settings.

Zitieren

APA

Liu, W., Huo, L., Jing, Y., Zhang, X., & Xie, J. (2026). MRACL: Multi-Reward Space Guided Adaptive Curriculum Reinforcement Learning for LLMs. Proceedings of the AAAI Conference on Artificial Intelligence, 40(44), 37663-37672. https://doi.org/10.1609/aaai.v40i44.41101

Vancouver

Liu W, Huo L, Jing Y, Zhang X, Xie J. MRACL: Multi-Reward Space Guided Adaptive Curriculum Reinforcement Learning for LLMs. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(44):37663-37672. doi:10.1609/aaai.v40i44.41101

BibTeX
@article{Liu2026,
  author = {Liu, Wenxuan and Huo, Liangyu and Jing, Yi and Zhang, Xiyuan and Xie, Jian},
  title = {MRACL: Multi-Reward Space Guided Adaptive Curriculum Reinforcement Learning for LLMs},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {44},
  pages = {37663--37672},
  doi = {10.1609/aaai.v40i44.41101}
}
Kostenlos starten →