Temporal Difference Learning with Experience Replay
Temporal-difference (TD) learning is widely regarded as one of the most popular algorithms in reinforcement learning (RL). Despite its widespread use, it has only been recently that researchers have begun to actively study its finite time behavior, including the finite time bound on mean squared error and sample complexity. On the empirical side, experience replay has been a key ingredient in the success of deep RL algorithms, but its theoretical effects on RL have yet to be fully understood. In this paper, we present a simple decomposition of the Markovian noise terms and provide finite-time error bounds for TD-learning with experience replay. Specifically, under the Markovian observation model, we demonstrate that for both the averaged iterate and final iterate cases, the error term induced by a constant step-size can be effectively controlled by the size of the replay buffer and the mini-batch sampled from the experience replay buffer.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Deep Reinforcement Learning with Quantum-inspired Experience Replay
In this paper, a novel training paradigm inspired by quantum computation is proposed for deep reinforcement learning (DRL) with experience replay. In contrast to traditional experience replay mechanism in DRL, the propos…
Atari GamesDeep Reinforcement LearningDiversityreinforcement-learning+2Revisiting Prioritized Experience Replay: A Value Perspective
Experience replay enables off-policy reinforcement learning (RL) agents to utilize past experiences to maximize the cumulative reward. Prioritized experience replay that weighs experiences by the magnitude of their tempo…
Atari GamesQ-LearningReinforcement Learning (RL)Experience Replay with Likelihood-free Importance Weights
The use of past experiences to accelerate temporal difference (TD) learning of value functions, or experience replay, is a key component in deep reinforcement learning. Prioritization or reweighting of important experien…
Deep Reinforcement LearningOpenAI Gymreinforcement-learningReinforcement Learning (RL)ROER: Regularized Optimal Experience Replay
Experience replay serves as a key component in the success of online reinforcement learning (RL). Prioritized experience replay (PER) reweights experiences by the temporal difference (TD) error empirically enhancing the …
continuous-controlContinuous ControlMuJoCoReinforcement Learning (RL)Return Dispersion as an Estimator of Learning Potential for Prioritized Level Replay
Prioritized Level Replay (PLR) has been shown to induce adaptive curricula that improve the sample-efficiency and generalization of reinforcement learning policies in environments featuring multiple tasks or levels. PLR …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)