Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement
Symmetry · 03.11.2026 · 18 (11) · 1849
DOI: 10.3390/sym18111849 ↗Kurz gefasst
Fachartikel in der Zeitschrift Symmetry, online erschienen am 03.11.2026. Verfasst von Zhong-Mei Gao. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: the National Key R&D Program of China, National Natural Science Foundation of China.
Abstract (Originalsprache)
Offline reinforcement learning aims to learn effective policies from fixed datasets without further interaction with the environment. However, limited behavior-policy coverage can lead to value-estimation errors and unreliable policy improvement, particularly when multiple critics provide conflicting update directions. To address these challenges, we propose Probability-Ratio-Constrained Q-Learning (PCR-QL), which integrates behavior-density modeling, conservative ensemble-based Q-learning, and probability-ratio-regularized policy extraction to control distributional shift. Building on this framework, we further introduce Consensus-Gated PCR-QL (CG-PCR-QL), which evaluates critic-specific actor gradients using shared states and policy noise and adaptively attenuates the ensemble-mean policy-improvement direction according to the weakest critic-gradient agreement. Theoretically, we establish results connecting ratio-constrained Bellman approximation, behavior-density estimation errors, policy-feasibility repair, and multi-critic gradient geometry, together with conditional guarantees for completed optimizer updates. Experiments on eleven Datasets for Deep Data-Driven Reinforcement Learning (D4RL) benchmark tasks compare nine policy-extraction rules across ten matched runs. CG-PCR-QL achieves a mean performance improvement of +6.32 (95% CI [5.14, 7.50]) over PCR-QL and a pooled interquartile-mean (IQM) improvement of +7.19 [5.63, 8.69]. Furthermore, an analysis of 2400 paired Adam-update events shows an increase of +13.0 [11.1, 14.9] percentage points in the frequency of simultaneous improvement across all critics. Additional ablations and diagnostics examine the effects of gradient consensus, behavior-density estimation, parameter sensitivity, and computational cost. These results highlight the effectiveness of integrating probability-ratio constraints with critic-gradient consensus for more reliable policy improvement in offline reinforcement learning.
Förderung
- the National Key R&D Program of China
- National Natural Science Foundation of China
Zitieren
APA
Gao, Z. (2026). Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement. Symmetry, 18(11), 1849. https://doi.org/10.3390/sym18111849
Vancouver
Gao Z. Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement. Symmetry. 2026;18(11):1849. doi:10.3390/sym18111849
BibTeX
@article{Gao2026,
author = {Gao, Zhong-Mei},
title = {Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement},
journal = {Symmetry},
year = {2026},
volume = {18},
number = {11},
pages = {1849},
doi = {10.3390/sym18111849}
}
EUVIMED