Contact Energy Based Hindsight Experience Prioritization
Multi-goal robot manipulation tasks with sparse rewards are difficult for reinforcement learning (RL) algorithms due to the inefficiency in collecting successful experiences. Recent algorithms such as Hindsight Experience Replay (HER) expedite learning by taking advantage of failed trajectories and replacing the desired goal with one of the achieved states so that any failed trajectory can be utilized as a contribution to learning. However, HER uniformly chooses failed trajectories, without taking into account which ones might be the most valuable for learning. In this paper, we address this problem and propose a novel approach Contact Energy Based Prioritization~(CEBP) to select the samples from the replay buffer based on rich information due to contact, leveraging the touch sensors in the gripper of the robot and object displacement. Our prioritization scheme favors sampling of contact-rich experiences, which are arguably the ones providing the largest amount of information. We evaluate our proposed approach on various sparse reward robotic tasks and compare them with the state-of-the-art methods. We show that our method surpasses or performs on par with those methods on robot manipulation tasks. Finally, we deploy the trained policy from our method to a real Franka robot for a pick-and-place task. We observe that the robot can solve the task successfully. The videos and code are publicly available at: https://erdiphd.github.io/HER_force
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement Learning (RL)Robot ManipulationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Energy-Based Hindsight Experience Prioritization
In Hindsight Experience Replay (HER), a reinforcement learning agent is trained by treating whatever it has achieved as virtual goals. However, in previous work, the experience was replayed at random, without considering…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
This paper proposes a method for prioritizing the replay experience referred to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight Experience Replay (HER) that generates hindsight goals based on un…
Regret Minimization Experience Replay in Off-Policy Reinforcement Learning
In reinforcement learning, experience replay stores past samples for further reuse. Prioritized sampling is a promising technique to better utilize these samples. Previous criteria of prioritization include TD error, rec…
MuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)Bootstrapped Hindsight Experience replay with Counterintuitive Prioritization
Goal-conditioned environments are known as sparse rewards tasks, in which the agent gains a positive reward only when it achieves the goal. Such an setting results in much difficulty for the agent to explore successful t…
Q-LearningCuriosity-Driven Experience Prioritization via Density Estimation
In Reinforcement Learning (RL), an agent explores the environment and collects trajectories into the memory buffer for later learning. However, the collected trajectories can easily be imbalanced with respect to the achi…
Density EstimationOpenAI Gymreinforcement-learningReinforcement Learning+2