Duality-Based Stochastic Policy Optimization for Estimation with Unknown Noise Covariances
Duality of control and estimation allows mapping recent advances in data-guided control to the estimation setup. This paper formalizes and utilizes such a mapping to consider learning the optimal (steady-state) Kalman gain when process and measurement noise statistics are unknown. Specifically, building on the duality between synthesizing optimal control and estimation gains, the filter design problem is formalized as direct policy learning. In this direction, the duality is used to extend existing theoretical guarantees of direct policy updates for Linear Quadratic Regulator (LQR) to establish global convergence of the Gradient Descent (GD) algorithm for the estimation problem--while addressing subtle differences between the two synthesis problems. Subsequently, a Stochastic Gradient Descent (SGD) approach is adopted to learn the optimal Kalman gain without the knowledge of noise covariances. The results are illustrated via several numerical examples.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Duality and Policy Evaluation in Distributionally Robust Bayesian Diffusion Control
We consider a Bayesian diffusion control problem of expected terminal utility maximization. The controller imposes a prior distribution on the unknown drift of an underlying diffusion. The Bayesian optimal control, track…
Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations
We consider large-scale Markov decision processes with an unknown cost function and address the problem of learning a policy from a finite set of expert demonstrations. We assume that the learner is not allowed to intera…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Reinforcement Learning via Fenchel-Rockafellar Duality
We review basic concepts of convex duality, focusing on the very general and supremely useful Fenchel-Rockafellar duality. We summarize how this duality may be applied to a variety of reinforcement learning (RL) settings…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
We study the online estimation of the optimal policy of a Markov decision process (MDP). We propose a class of Stochastic Primal-Dual (SPD) methods which exploit the inherent minimax duality of Bellman equations. The SPD…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Offline Policy Optimization with Variance Regularization
Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributiona…
continuous-controlContinuous ControlOffline RLReinforcement Learning (RL)