EUVIMEDEUVIMED entdecken
Crossref

Revisiting Overestimation Bias of Q-learning: Breaking Bias Propagation Chains Does Well

Hong Xie, Tao Tan, Defu Lian, Pu Li

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026 · 1-12

DOI: 10.1109/tpami.2026.3750927 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. Verfasst von Hong Xie und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper revisits the overestimation bias of Q-learning from a new perspective, i.e., the breaking bias propagation chains. We make five-fold contributions. First, we analyze the estimation bias propagation chains of Q-learning, and find that the bias propagated from previous steps dominates the maximum Q-value estimation bias and slows the convergence speed, instead of the current bias. Second, we propose a novel positive-negative bias alternating algorithm called Alternating Q-learning (AQ). It breaks the unidirectional estimation bias propagation chains via alternately executing Q-learning and Double Q-learning. We show theoretically that there exist suitable alternating parameters to eliminate the propagation bias. Third, we design an adaptive alternating strategy for AQ, obtaining Adaptive Alternating Q-learning (AdaAQ). It applies a softmax strategy with the absolute value of TD error to choose Q-learning or Double Q-learning for each state-action pair. Fourth, we extend AQ and AdaAQ to the large-scale settings with function approximation, i.e., including both discrete- and continuous-action Deep Reinforcement Learning (DRL). Fifth, both discrete-and continuous-action DRL experiments show that our method outperforms several state-of-the-art (SOTA) baselines drastically; tabular MDP experiments reveal fundamental insights into why our method can achieve superior performance.

Zitieren

APA

Xie, H., Tan, T., Lian, D., & Li, P. (2026). Revisiting Overestimation Bias of Q-learning: Breaking Bias Propagation Chains Does Well. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1-12. https://doi.org/10.1109/tpami.2026.3750927

Vancouver

Xie H, Tan T, Lian D, Li P. Revisiting Overestimation Bias of Q-learning: Breaking Bias Propagation Chains Does Well. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026:1-12. doi:10.1109/tpami.2026.3750927

BibTeX
@article{Xie2026,
  author = {Xie, Hong and Tan, Tao and Lian, Defu and Li, Pu},
  title = {Revisiting Overestimation Bias of Q-learning: Breaking Bias Propagation Chains Does Well},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  pages = {1--12},
  doi = {10.1109/tpami.2026.3750927}
}
Kostenlos starten →