EUVIMEDEUVIMED entdecken
Crossref

Reinforcement Learning with Function Approximation: From Linear to Nonlinear

Jihao Long, Jiequn Han

Journal of Machine Learning · 2023

DOI: 10.4208/jml.230105 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Machine Learning, 2023. Verfasst von Jihao Long und 1 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 61 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Function approximation has been an indispensable component in modern reinforcement learning algorithms designed to tackle problems with large state spaces in high dimensions. This paper reviews recent results on error analysis for these reinforcement learning algorithms in linear or nonlinear approximation settings, emphasizing approximation error and estimation error/sample complexity. We discuss various properties related to approximation error and present concrete conditions on transition probability and reward function under which these properties hold true. Sample complexity analysis in reinforcement learning is more complicated than in supervised learning, primarily due to the distribution mismatch phenomenon. With assumptions on the linear structure of the problem, numerous algorithms in the literature achieve polynomial sample complexity with respect to the number of features, episode length, and accuracy, although the minimax rate has not been achieved yet. These results rely on the $L^∞$ and UCB estimation of estimation error, which can handle the distribution mismatch phenomenon. The problem and analysis become substantially more challenging in the setting of nonlinear function approximation, as both $L^∞$ and UCB estimation are inadequate for bounding the error with a favorable rate in high dimensions. We discuss additional assumption necessary to address the distribution mismatch and derive meaningful results for nonlinear RL problems.

Zitieren

APA

Long, J., & Han, J. (2023). Reinforcement Learning with Function Approximation: From Linear to Nonlinear. Journal of Machine Learning. https://doi.org/10.4208/jml.230105

Vancouver

Long J, Han J. Reinforcement Learning with Function Approximation: From Linear to Nonlinear. Journal of Machine Learning. 2023. doi:10.4208/jml.230105

BibTeX
@article{Long2023,
  author = {Long, Jihao and Han, Jiequn},
  title = {Reinforcement Learning with Function Approximation: From Linear to Nonlinear},
  journal = {Journal of Machine Learning},
  year = {2023},
  doi = {10.4208/jml.230105}
}
Kostenlos starten →