Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems
We study derivative-free methods for policy optimization over the class of linear policies. We focus on characterizing the convergence rate of these methods when applied to linear-quadratic systems, and study various settings of driving noise and reward feedback. We show that these methods provably converge to within any pre-specified tolerance of the optimal policy with a number of zero-order evaluations that is an explicit polynomial of the error tolerance, dimension, and curvature properties of the problem. Our analysis reveals some interesting differences between the settings of additive driving noise and random initialization, as well as the settings of one-point and two-point reward feedback. Our theory is corroborated by extensive simulations of derivative-free methods on these systems. Along the way, we derive convergence rates for stochastic zero-order optimization algorithms when applied to a certain class of non-convex problems.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Model-Free $μ$-Synthesis: A Nonsmooth Optimization Perspective
In this paper, we revisit model-free policy search on an important robust control benchmark, namely $\mu$-synthesis. In the general output-feedback setting, there do not exist convex formulations for this problem, and he…
modelComplexity of Derivative-Free Policy Optimization for Structured $\mathcal{H}_\infty$ Control
The applications of direct policy search in reinforcement learning and continuous control have received increasing attention. In this work, we present novel theoretical results on the complexity of derivative-free policy…
Efficient Hyperparameter Tuning with Dynamic Accuracy Derivative-Free Optimization
Many machine learning solutions are framed as optimization problems which rely on good hyperparameters. Algorithms for tuning these hyperparameters usually assume access to exact solutions to the underlying learning prob…
ZOOpt: Toolbox for Derivative-Free Optimization
Recent advances in derivative-free optimization allow efficient approximation of the global-optimal solutions of sophisticated functions, such as functions with many local optima, non-differentiable and non-continuous fu…
BIG-bench Machine LearningDistributed OptimizationAdversarial Imitation Learning via Random Search
Developing agents that can perform challenging complex tasks is the goal of reinforcement learning. The model-free reinforcement learning has been considered as a feasible solution. However, the state of the art research…
Computational EfficiencyDeep Reinforcement LearningImitation LearningMuJoCo+3