Accuracy of Discretely Sampled Stochastic Policies in Continuous-Time Reinforcement Learning
SIAM Journal on Control and Optimization · 30.06.2026 · 64 (3) · 1889-1929
DOI: 10.1137/25m1742199 ↗Kurz gefasst
Fachartikel in der Zeitschrift SIAM Journal on Control and Optimization, online erschienen am 11.06.2026. Verfasst von Yanwei Jia und 2 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: Chinese University of Hong Kong, Research Grants Council, University Grants Committee, International Research Lab.
Abstract (Originalsprache)
Stochastic policies (also known as relaxed controls) are widely used in continuous-time reinforcement learning algorithms. However, executing a stochastic policy and evaluating its performance in a continuous-time environment remain open challenges. This work introduces and rigorously analyzes a policy execution framework that samples actions from a stochastic policy at discrete time points and implements them as piecewise constant controls. We prove that as the sampling mesh size tends to zero, the controlled state process converges weakly to the dynamics with coefficients aggregated according to the stochastic policy. We explicitly quantify the convergence rate based on the regularity of the coefficients and establish an optimal first-order convergence rate for sufficiently regular coefficients. Additionally, we prove a [Formula: see text]-order weak convergence rate that holds uniformly over the sampling noise with high probability and establish a [Formula: see text]-order pathwise convergence for each realization of the system noise in the absence of volatility control. Building on these results, we analyze the bias and variance of various policy evaluation and policy gradient estimators based on discrete-time observations. Our results provide theoretical justification for the exploratory stochastic control framework in [H. Wang, T. Zariphopoulou, and X. Y. Zhou, J. Mach. Learn. Res., 21 (2020), pp. 1–34].
Förderung
- Chinese University of Hong Kong
- Research Grants Council, University Grants Committee
- International Research Lab
Zitieren
APA
Jia, Y., Ouyang, D., & Zhang, Y. (2026). Accuracy of Discretely Sampled Stochastic Policies in Continuous-Time Reinforcement Learning. SIAM Journal on Control and Optimization, 64(3), 1889-1929. https://doi.org/10.1137/25m1742199
Vancouver
Jia Y, Ouyang D, Zhang Y. Accuracy of Discretely Sampled Stochastic Policies in Continuous-Time Reinforcement Learning. SIAM Journal on Control and Optimization. 2026;64(3):1889-1929. doi:10.1137/25m1742199
BibTeX
@article{Jia2026,
author = {Jia, Yanwei and Ouyang, Du and Zhang, Yufei},
title = {Accuracy of Discretely Sampled Stochastic Policies in Continuous-Time Reinforcement Learning},
journal = {SIAM Journal on Control and Optimization},
year = {2026},
volume = {64},
number = {3},
pages = {1889--1929},
doi = {10.1137/25m1742199}
}
EUVIMED