EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement

Zhong-Mei Gao

Symmetry · 03.11.2026 · 18 (11) · 1849

DOI: 10.3390/sym18111849 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Symmetry, online erschienen am 03.11.2026. Verfasst von Zhong-Mei Gao. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: the National Key R&D Program of China, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Offline reinforcement learning aims to learn effective policies from fixed datasets without further interaction with the environment. However, limited behavior-policy coverage can lead to value-estimation errors and unreliable policy improvement, particularly when multiple critics provide conflicting update directions. To address these challenges, we propose Probability-Ratio-Constrained Q-Learning (PCR-QL), which integrates behavior-density modeling, conservative ensemble-based Q-learning, and probability-ratio-regularized policy extraction to control distributional shift. Building on this framework, we further introduce Consensus-Gated PCR-QL (CG-PCR-QL), which evaluates critic-specific actor gradients using shared states and policy noise and adaptively attenuates the ensemble-mean policy-improvement direction according to the weakest critic-gradient agreement. Theoretically, we establish results connecting ratio-constrained Bellman approximation, behavior-density estimation errors, policy-feasibility repair, and multi-critic gradient geometry, together with conditional guarantees for completed optimizer updates. Experiments on eleven Datasets for Deep Data-Driven Reinforcement Learning (D4RL) benchmark tasks compare nine policy-extraction rules across ten matched runs. CG-PCR-QL achieves a mean performance improvement of +6.32 (95% CI [5.14, 7.50]) over PCR-QL and a pooled interquartile-mean (IQM) improvement of +7.19 [5.63, 8.69]. Furthermore, an analysis of 2400 paired Adam-update events shows an increase of +13.0 [11.1, 14.9] percentage points in the frequency of simultaneous improvement across all critics. Additional ablations and diagnostics examine the effects of gradient consensus, behavior-density estimation, parameter sensitivity, and computational cost. These results highlight the effectiveness of integrating probability-ratio constraints with critic-gradient consensus for more reliable policy improvement in offline reinforcement learning.

Förderung

  • the National Key R&D Program of China
  • National Natural Science Foundation of China

Zitieren

APA

Gao, Z. (2026). Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement. Symmetry, 18(11), 1849. https://doi.org/10.3390/sym18111849

Vancouver

Gao Z. Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement. Symmetry. 2026;18(11):1849. doi:10.3390/sym18111849

BibTeX
@article{Gao2026,
  author = {Gao, Zhong-Mei},
  title = {Ratio-Constrained Offline Reinforcement Learning: Consensus-Gated Policy Improvement via Critic Gradient Agreement},
  journal = {Symmetry},
  year = {2026},
  volume = {18},
  number = {11},
  pages = {1849},
  doi = {10.3390/sym18111849}
}
Kostenlos starten →