EUVIMEDEUVIMED entdecken
Crossref

Zeroth-Order Actor–Critic: An Evolutionary Framework for Sequential Decision Problems

Yuheng Lei, Yao Lyu, Guojian Zhan, Tao Zhang, Jiangtao Li, Jianyu Chen, Shengbo Eben Li, Sifa Zheng

IEEE Transactions on Evolutionary Computation · 2025-04 · 29 (2) · 555-569

DOI: 10.1109/tevc.2025.3529503 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Evolutionary Computation, 2025-04. Verfasst von Yuheng Lei und 7 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 82 Literaturangaben. Gefördert von: National Key Research and Development Program of China, Tsinghua-Toyota Joint Research Fund, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Evolutionary algorithms (EAs) have shown promise in solving sequential decision problems (SDPs) by simplifying them to static optimization problems and searching for the optimal policy parameters in a zeroth-order way. Despite their versatility, EAs often suffer from high sample complexity due to neglecting underlying temporal structures. In contrast, reinforcement learning (RL) methods typically formulate SDPs as Markov decision process (MDP). Although more sample efficient than EAs, RL methods are restricted to differentiable policies and prone to getting stuck in local optima. To address these issues, we propose a novel evolutionary framework zeroth-order actor-critic (ZOAC). We propose to use stepwise exploration in parameter space and theoretically derive the zeroth-order policy gradient. We further utilize the actor-critic architecture to effectively leverage the Markov property of SDPs and reduce the variance of gradient estimators. In each iteration, ZOAC collects trajectories with parameter space exploration, and alternates between first-order policy evaluation (PEV) and zeroth-order policy improvement (PIM). We evaluate the effectiveness of ZOAC on a challenging multilane driving task optimizing the parameters in a rule-based, nondifferentiable driving policy that consists of three submodules: 1) behavior selection; 2) path planning; and 3) trajectory tracking. We also compare it with gradient-based RL methods on three Gymnasium tasks, optimizing neural network policies with thousands of parameters. Experimental results demonstrate the strong capability of ZOAC in solving SDPs. ZOAC significantly outperforms EAs that treat the problem as static optimization and matches the performance of gradient-based RL methods even without first-order information, in terms of total average return across tasks.

Förderung

  • National Key Research and Development Program of China
  • Tsinghua-Toyota Joint Research Fund
  • National Natural Science Foundation of China

Zitieren

APA

Lei, Y., Lyu, Y., Zhan, G., Zhang, T., Li, J., Chen, J., Li, S. E., & Zheng, S. (2025). Zeroth-Order Actor–Critic: An Evolutionary Framework for Sequential Decision Problems. IEEE Transactions on Evolutionary Computation, 29(2), 555-569. https://doi.org/10.1109/tevc.2025.3529503

Vancouver

Lei Y, Lyu Y, Zhan G, Zhang T, Li J, Chen J, et al. Zeroth-Order Actor–Critic: An Evolutionary Framework for Sequential Decision Problems. IEEE Transactions on Evolutionary Computation. 2025;29(2):555-569. doi:10.1109/tevc.2025.3529503

BibTeX
@article{Lei2025,
  author = {Lei, Yuheng and Lyu, Yao and Zhan, Guojian and Zhang, Tao and Li, Jiangtao and Chen, Jianyu and Li, Shengbo Eben and Zheng, Sifa},
  title = {Zeroth-Order Actor–Critic: An Evolutionary Framework for Sequential Decision Problems},
  journal = {IEEE Transactions on Evolutionary Computation},
  year = {2025},
  volume = {29},
  number = {2},
  pages = {555--569},
  doi = {10.1109/tevc.2025.3529503}
}
Kostenlos starten →