EUVIMEDEUVIMED entdecken
Crossref

Frozen-State Value Iteration: Faster Reinforcement Learning by Freezing Slow States

Yijia Wang, Daniel R. Jiang

Management Science · 2026-11 · 72 (11) · 9545-9563

DOI: 10.1287/mnsc.2023.00012 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Management Science, 2026-11. Verfasst von Yijia Wang und 1 weiteren Autor:innen. Die Arbeit nennt 77 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We study infinite-horizon Markov decision processes (MDPs) with fast–slow structure, in which some state variables evolve rapidly (fast states), whereas others change more gradually (slow states). This structure commonly arises in practice when decisions must be made at high frequencies over long horizons and when slowly changing information still plays a critical role in determining optimal actions. Examples include inventory control under slowly changing demand indicators or dynamic pricing with gradually shifting consumer behavior. Modeling the problem at the natural decision frequency leads to MDPs with discount factors close to one, making them computationally challenging. We propose a novel approximation strategy that freezes slow states during phases of lower level planning and subsequently applies value iteration to an auxiliary upper level MDP that evolves on a slower timescale. Freezing states for short periods of time leads to easier to solve lower level problems, whereas a slower upper level timescale allows for a more favorable discount factor. On the theoretical side, we analyze the regret incurred by our frozen-state approach, and this leads to simple insights on how to trade off regret versus computational cost. Empirically, we benchmark our new frozen-state methods on three domains: (i) inventory control with fixed order costs, (ii) a grid world problem with spatial tasks, and (iii) dynamic pricing with reference price effects. We demonstrate that the new methods produce high-quality policies with significantly less computation, and we show that simply omitting slow states is often a poor heuristic. This paper was accepted by J. George Shanthikumar, data science. Funding: This research is based upon work supported by the U.S. National Science Foundation [Grant 1807536]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2023.00012 .

Zitieren

APA

Wang, Y., & Jiang, D. R. (2026). Frozen-State Value Iteration: Faster Reinforcement Learning by Freezing Slow States. Management Science, 72(11), 9545-9563. https://doi.org/10.1287/mnsc.2023.00012

Vancouver

Wang Y, Jiang DR. Frozen-State Value Iteration: Faster Reinforcement Learning by Freezing Slow States. Management Science. 2026;72(11):9545-9563. doi:10.1287/mnsc.2023.00012

BibTeX
@article{Wang2026,
  author = {Wang, Yijia and Jiang, Daniel R.},
  title = {Frozen-State Value Iteration: Faster Reinforcement Learning by Freezing Slow States},
  journal = {Management Science},
  year = {2026},
  volume = {72},
  number = {11},
  pages = {9545--9563},
  doi = {10.1287/mnsc.2023.00012}
}
Kostenlos starten →