Average Reward Reinforcement Learning with Monotonic Policy Improvement
In continuing control tasks, an agent’s average reward per time step is a more natural performance measure compared to the commonly used discounting framework as it can better capture an agent’s long-term behavior. We derive a novel lower bound on the difference of the average rewards for two policies, where the lower bound depends on the average divergence between the policies. We show that previous work based on the discounted return (Schulman et al., 2015; Achiam et al.,2017) result in a trivial lower bound in the average reward setting. We develop an iterative procedure based on our lower bound which produces a sequence of monotonically improved policies for the average reward criterion. When combined with deep reinforcement learning methods, the procedure leads to scalable and efficient algorithms aimed at maximizing an agent’s average reward performance. Empirically, we demonstrate the efficacy of our algorithms through a series of high-dimensional control tasks with long time horizons and show that discounting can lead to unsatisfactory performance on continuing control tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
On-Policy Deep Reinforcement Learning for the Average-Reward Criterion
We develop theory and algorithms for average-reward on-policy Reinforcement Learning (RL). We first consider bounding the difference of the long-term average reward for two policies. We show that previous work based on t…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but…
Reinforcement LearningPerformance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithms
Many policy-based reinforcement learning (RL) algorithms can be viewed as instantiations of approximate policy iteration (PI), i.e., where policy improvement and policy evaluation are both performed approximately. In app…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning
In this paper, we propose an off-policy deep reinforcement learning (DRL) method utilizing the average reward criterion. While most existing DRL methods employ the discounted reward criterion, this can potentially lead t…
Feasible Policy Iteration for Safe Reinforcement Learning
Safety is the priority concern when applying reinforcement learning (RL) algorithms to real-world control problems. While policy iteration provides a fundamental algorithm for standard RL, an analogous theoretical algori…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning