USHER: Unbiased Sampling for Hindsight Experience Replay
Dealing with sparse rewards is a long-standing challenge in reinforcement learning (RL). Hindsight Experience Replay (HER) addresses this problem by reusing failed trajectories for one goal as successful trajectories for another. This allows for both a minimum density of reward and for generalization across multiple goals. However, this strategy is known to result in a biased value function, as the update rule underestimates the likelihood of bad outcomes in a stochastic environment. We propose an asymptotically unbiased importance-sampling-based algorithm to address this problem without sacrificing performance on deterministic environments. We show its effectiveness on a range of robotic systems, including challenging high dimensional stochastic environments.
Code (1)
Tasks
reinforcement-learningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
This paper proposes a method for prioritizing the replay experience referred to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight Experience Replay (HER) that generates hindsight goals based on un…
ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay
Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful…
continuous-controlContinuous ControlDeep Reinforcement LearningReinforcement Learning+1Cluster-based Sampling in Hindsight Experience Replay for Robotic Tasks (Student Abstract)
In multi-goal reinforcement learning with a sparse binary reward, training agents is particularly challenging, due to a lack of successful experiences. To solve this problem, hindsight experience replay (HER) generates s…
ClusteringMulti-Goal Reinforcement LearningOpenAI GymHindsight Curriculum Generation Based Multi-Goal Experience Replay
In multi-goal tasks with sparse rewards, it is challenging to learn from tons of experiences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown…
Reinforcement Learning (RL)Introspective Experience Replay: Look Back When Surprised
In reinforcement learning (RL), experience replay-based sampling techniques play a crucial role in promoting convergence by eliminating spurious correlations. However, widely used methods such as uniform experience repla…
Q-Learningreinforcement-learningReinforcement Learning (RL)