Model-augmented Prioritized Experience Replay

2021-09-29 · ICLR 2022 4 · Youngmin Oh, Jinwoo Shin, Eunho Yang, Sung Ju Hwang

Experience replay is an essential component in off-policy model-free reinforcement learning (MfRL). Due to its effectiveness, various methods for calculating priority scores on experiences have been proposed for sampling. Since critic networks are crucial to policy learning, TD-error, directly correlated to $Q$-values, is one of the most frequently used features to compute the scores. However, critic networks often under- or overestimate $Q$-values, so it is often ineffective to learn to predict $Q$-values by sampled experiences based heavily on TD-error. Accordingly, it is valuable to find auxiliary features, which positively support TD-error in calculating the scores for efficient sampling. Motivated by this, we propose a novel experience replay method, which we call model-augmented priority experience replay (MaPER), that employs new learnable features driven from components in model-based RL (MbRL) to calculate the scores on experiences. The proposed MaPER brings the effect of curriculum learning for predicting $Q$-values better by the critic network with negligible memory and computational overhead compared to the vanilla PER. Indeed, our experimental results on various tasks demonstrate that MaPER can significantly improve the performance of the state-of-the-art off-policy MfRL and MbRL which includes off-policy MfRL algorithms in its policy optimization procedure.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

model

Methods 이 논문이 사용한 방법론

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Double Prioritized State Recycled Experience Replay

2020-07-08 · Fanchen Bu, Dong Eui Chang

Experience replay enables online reinforcement learning agents to store and reuse the previous experiences of interacting with the environment. In the original method, the experiences are sampled and replayed uniformly a…

Atari Games

A novel DDPG method with prioritized experience replay

2017-10-01 · IEEE International Conference on Systems, Man and Cybernetics (SMC) 2017 10 · Yuenan Hou, Lifeng Liu, Qing Wei, Xudong Xu 외

Recently, a state-of-the-art algorithm, called deep deterministic policy gradient (DDPG), has achieved good performance in many continuous control tasks in the MuJoCo simulator. To further improve the efficiency of the e…

continuous-control Continuous Control MuJoCo OpenAI Gym

Prioritized Experience Replay

2015-11-18 · Tom Schaul, John Quan, Ioannis Antonoglou, David Silver

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply r…

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Improving Experience Replay with Successor Representation

2021-11-29 · Yizhi Yuan, Marcelo G Mattar

Prioritized experience replay is a reinforcement learning technique whereby agents speed up learning by replaying useful past experiences. This usefulness is quantified as the expected gain from replaying the experience,…

Atari Games

Prioritized Sequence Experience Replay

2019-05-25 · Marc Brittain, Josh Bertram, Xuxi Yang, Peng Wei

Experience replay is widely used in deep reinforcement learning algorithms and allows agents to remember and learn from experiences from the past. In an effort to learn more efficiently, researchers proposed prioritized …

Deep Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1