When Q-Learning fails: unstable behavior for infinite state spaces
ACM SIGMETRICS Performance Evaluation Review · 31.03.2026 · 53 (4) · 79-83
DOI: 10.1145/3797823.3797849 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM SIGMETRICS Performance Evaluation Review, online erschienen am 31.03.2026. Verfasst von U. Ayestaa und 3 weiteren Autor:innen. Die Arbeit nennt 9 Literaturangaben.
Abstract (Originalsprache)
The Q-learning algorithm is well known for its convergence guarantees to the optimal policy in finite-state environments. In this paper, we investigate its limitations in countable infinite state spaces – a setting common in real-world problems. To this end, we introduce a simple queueing model, based on a load balancing problem, with a countably infinite state space. In this model, a dispatcher assigns incoming jobs to one of two queues by choosing between two possible actions: ''red'' and ''green''. The ''red'' action leads to transient behavior, whereas the ''green'' action ensures stability. Our main result shows that, under certain parameter conditions, Q-learning exhibits instability and fails to converge to the optimal policy. Our findings reveal a critical gap in the theoretical understanding of model-free Reinforcement Learning methods in infinite domains. Numerical experiments illustrate that the transience also occurs with decreasing stepsizes that satisfy the usual Robbins-Monro conditions.
Zitieren
APA
Ayestaa, U., Foss, S., Jonckheere, M., & Puricellia, V. (2026). When Q-Learning fails: unstable behavior for infinite state spaces. ACM SIGMETRICS Performance Evaluation Review, 53(4), 79-83. https://doi.org/10.1145/3797823.3797849
Vancouver
Ayestaa U, Foss S, Jonckheere M, Puricellia V. When Q-Learning fails: unstable behavior for infinite state spaces. ACM SIGMETRICS Performance Evaluation Review. 2026;53(4):79-83. doi:10.1145/3797823.3797849
BibTeX
@article{Ayestaa2026,
author = {Ayestaa, U. and Foss, S. and Jonckheere, M. and Puricellia, V.},
title = {When Q-Learning fails: unstable behavior for infinite state spaces},
journal = {ACM SIGMETRICS Performance Evaluation Review},
year = {2026},
volume = {53},
number = {4},
pages = {79--83},
doi = {10.1145/3797823.3797849}
}
EUVIMED