Improving Learning from Demonstrations by Learning from Experience
How to make imitation learning more general when demonstrations are relatively limited has been a persistent problem in reinforcement learning (RL). Poor demonstrations lead to narrow and biased date distribution, non-Markovian human expert demonstration makes it difficult for the agent to learn, and over-reliance on sub-optimal trajectories can make it hard for the agent to improve its performance. To solve these problems we propose a new algorithm named TD3fG that can smoothly transition from learning from experts to learning from experience. Our algorithm achieves good performance in the MUJOCO environment with limited and sub-optimal demonstrations. We use behavior cloning to train the network as a reference action generator and utilize it in terms of both loss function and exploration noise. This innovation can help agents extract a priori knowledge from demonstrations while reducing the detrimental effects of the poor Markovian properties of the demonstrations. It has a better performance compared to the BC+ fine-tuning and DDPGfD approach, especially when the demonstrations are relatively limited. We call our method TD3fG meaning TD3 from a generator.
Code (0)
등록된 구현이 없습니다.
Tasks
Imitation LearningMuJoCoReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Imitation Learning from Demonstrations
Despite the numerous breakthroughs achieved with Reinforcement Learning (RL), solving environments with sparse rewards remains a challenging task that requires sophisticated exploration. Learning from Demonstrations (LfD…
Imitation LearningReinforcement Learning (RL)XPACE: Joint World and Action Modeling from Heterogeneous Experience
A general-purpose robot needs to draw on diverse experience, choose actions, and anticipate how those actions will change the world. We introduce XPACE, a unified embodied world model that serves as both a world action m…
Video PredictionWatch, Try, Learn: Meta-Learning from Demonstrations and Reward
Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising a…
Imitation LearningMeta-LearningMeta Reinforcement LearningReinforcement LearningWatch, Try, Learn: Meta-Learning from Demonstrations and Rewards
Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising a…
Imitation LearningMeta-LearningMeta Reinforcement LearningAccelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble
Deep reinforcement learning (DRL) provides a new way to generate robot control policy. However, the process of training control policy requires lengthy exploration, resulting in a low sample efficiency of reinforcement l…
continuous-controlContinuous ControlDeep Reinforcement LearningImitation Learning+4