Globally Convergent Policy Search over Dynamic Filters for Output Estimation
We introduce the first direct policy search algorithm which provably converges to the globally optimal $\textit{dynamic}$ filter for the classical problem of predicting the outputs of a linear dynamical system, given noisy, partial observations. Despite the ubiquity of partial observability in practice, theoretical guarantees for direct policy search algorithms, one of the backbones of modern reinforcement learning, have proven difficult to achieve. This is primarily due to the degeneracies which arise when optimizing over filters that maintain internal state. In this paper, we provide a new perspective on this challenging problem based on the notion of $\textit{informativity}$, which intuitively requires that all components of a filter's internal state are representative of the true state of the underlying dynamical system. We show that informativity overcomes the aforementioned degeneracy. Specifically, we propose a $\textit{regularizer}$ which explicitly enforces informativity, and establish that gradient descent on this regularized objective - combined with a ``reconditioning step'' - converges to the globally optimal cost a $\mathcal{O}(1/T)$ rate. Our analysis relies on several new results which may be of independent interest, including a new framework for analyzing non-convex gradient descent via convex reformulation, and novel bounds on the solution to linear Lyapunov equations in terms of (our quantitative measure of) informativity.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Globally exponentially convergent observer for systems evolving on matrix Lie groups
We propose a globally exponentially convergent observer for the dynamical system evolving on matrix Lie groups with bounded velocity with unknown bound. We design the observer in the ambient Euclidean space and show expo…
TranslationConvergent dynamics of optimal nonlinear damping control
Following Demidovich's concept and definition of convergent systems, we analyze the optimal nonlinear damping control, recently proposed [1] for the second-order systems. Targeting the problem of output regulation, corre…
Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control
This paper proposes a novel robust reinforcement learning framework for discrete-time linear systems with model mismatch that may arise from the sim-to-real gap. A key strategy is to invoke advanced techniques from contr…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Globally Convergent Multilevel Training of Deep Residual Networks
We propose a globally convergent multilevel training method for deep residual networks (ResNets). The devised method can be seen as a novel variant of the recursive multilevel trust-region (RMTR) method, which operates i…
Observability is Sufficient for the Design of Globally Exponentially Convergent State Observers for State-affine Nonlinear Systems
In this paper we are interested in the problem of state observation of state-affine nonlinear systems. Our main contribution is to propose a globally exponentially convergent observer that requires only the necessary ass…