Reward-Weighted Regression Converges to a Global Optimum
Reward-Weighted Regression (RWR) belongs to a family of widely known iterative Reinforcement Learning algorithms based on the Expectation-Maximization framework. In this family, learning at each iteration consists of sampling a batch of trajectories using the current policy and fitting a new policy to maximize a return-weighted log-likelihood of actions. Although RWR is known to yield monotonic improvement of the policy under certain circumstances, whether and under which conditions RWR converges to the optimal policy have remained open questions. In this paper, we provide for the first time a proof that RWR converges to a global optimum when no function approximation is used, in a general compact setting. Furthermore, for the simpler case with finite state and action spaces we prove R-linear convergence of the state-value function to the optimum.
Code (1)
Tasks
regressionReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Understanding Learning Invariance in Deep Linear Networks
Equivariant and invariant machine learning models exploit symmetries and structural patterns in data to improve sample efficiency. While empirical studies suggest that data-driven methods such as regularization and data …
Data AugmentationNoisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization
Numerous empirical evidences have corroborated the importance of noise in nonconvex optimization problems. The theory behind such empirical observations, however, is still largely unknown. This paper studies this fundame…
Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
Despite the wide applications of Adam in reinforcement learning (RL), the theoretical convergence of Adam-type RL algorithms has not been established. This paper provides the first such convergence analysis for two funda…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)A Deterministic Global Optimization Method for Variational Inference
Variational inference methods for latent variable statistical models have gained popularity because they are relatively fast, can handle large data sets, and have deterministic convergence guarantees. However, in practic…
global-optimizationVariational InferenceNeural Temporal-Difference Learning Converges to Global Optima
Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coup…
Deep Reinforcement LearningQ-LearningReinforcement LearningReinforcement Learning (RL)