EUVIMEDEUVIMED entdecken
Crossref

When Q-Learning fails: unstable behavior for infinite state spaces

U. Ayestaa, S. Foss, M. Jonckheere, V. Puricellia

ACM SIGMETRICS Performance Evaluation Review · 31.03.2026 · 53 (4) · 79-83

DOI: 10.1145/3797823.3797849 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM SIGMETRICS Performance Evaluation Review, online erschienen am 31.03.2026. Verfasst von U. Ayestaa und 3 weiteren Autor:innen. Die Arbeit nennt 9 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Q-learning algorithm is well known for its convergence guarantees to the optimal policy in finite-state environments. In this paper, we investigate its limitations in countable infinite state spaces – a setting common in real-world problems. To this end, we introduce a simple queueing model, based on a load balancing problem, with a countably infinite state space. In this model, a dispatcher assigns incoming jobs to one of two queues by choosing between two possible actions: ''red'' and ''green''. The ''red'' action leads to transient behavior, whereas the ''green'' action ensures stability. Our main result shows that, under certain parameter conditions, Q-learning exhibits instability and fails to converge to the optimal policy. Our findings reveal a critical gap in the theoretical understanding of model-free Reinforcement Learning methods in infinite domains. Numerical experiments illustrate that the transience also occurs with decreasing stepsizes that satisfy the usual Robbins-Monro conditions.

Zitieren

APA

Ayestaa, U., Foss, S., Jonckheere, M., & Puricellia, V. (2026). When Q-Learning fails: unstable behavior for infinite state spaces. ACM SIGMETRICS Performance Evaluation Review, 53(4), 79-83. https://doi.org/10.1145/3797823.3797849

Vancouver

Ayestaa U, Foss S, Jonckheere M, Puricellia V. When Q-Learning fails: unstable behavior for infinite state spaces. ACM SIGMETRICS Performance Evaluation Review. 2026;53(4):79-83. doi:10.1145/3797823.3797849

BibTeX
@article{Ayestaa2026,
  author = {Ayestaa, U. and Foss, S. and Jonckheere, M. and Puricellia, V.},
  title = {When Q-Learning fails: unstable behavior for infinite state spaces},
  journal = {ACM SIGMETRICS Performance Evaluation Review},
  year = {2026},
  volume = {53},
  number = {4},
  pages = {79--83},
  doi = {10.1145/3797823.3797849}
}
Kostenlos starten →