Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations

2020-04-01 · Zhuangdi Zhu, Kaixiang Lin, Bo Dai, Jiayu Zhou

Model-free deep reinforcement learning (RL) has demonstrated its superiority on many complex sequential decision-making problems. However, heavy dependence on dense rewards and high sample-complexity impedes the wide adoption of these methods in real-world scenarios. On the other hand, imitation learning (IL) learns effectively in sparse-rewarded tasks by leveraging the existing expert demonstrations. In practice, collecting a sufficient amount of expert demonstrations can be prohibitively expensive, and the quality of demonstrations typically limits the performance of the learning policy. In this work, we propose Self-Adaptive Imitation Learning (SAIL) that can achieve (near) optimal performance given only a limited number of sub-optimal demonstrations for highly challenging sparse reward tasks. SAIL bridges the advantages of IL and RL to reduce the sample complexity substantially, by effectively exploiting sup-optimal demonstrations and efficiently exploring the environment to surpass the demonstrated performance. Extensive empirical results show that not only does SAIL significantly improve the sample-efficiency but also leads to much better final performance across different continuous control tasks, comparing to the state-of-the-art.

📄 PDF Abstract BibTeX arXiv:2004.00530

Code (1)

illidanlab/sail tf

Tasks

continuous-control Continuous Control Decision MakingDeep Reinforcement LearningImitation LearningReinforcement Learning Reinforcement Learning (RL)Sequential Decision Making

Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks

2024-07-31 · David Valencia, Henry Williams, Yuning Xing, Trevor Gee 외

Reinforcement Learning (RL) has been widely used to solve tasks where the environment consistently provides a dense reward value. However, in real-world scenarios, rewards can often be poorly defined or sparse. Auxiliary…

continuous-control Continuous Control Deep Reinforcement Learning Efficient Exploration+1

Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation

2021-06-23 · CVPR 2022 1 · Stephen James, Kentaro Wada, Tristan Laidlow, Andrew J. Davison

We present a coarse-to-fine discretisation method that enables the use of discrete reinforcement learning approaches in place of unstable and data-inefficient actor-critic methods in continuous robotics domains. This app…

Continuous Control Q-Learning Robot Manipulation Translation

Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation

2021-05-31 · Stephen James, Andrew J. Davison

Despite the success of reinforcement learning methods, they have yet to have their breakthrough moment when applied to a broad range of robotic manipulation tasks. This is partly due to the fact that reinforcement learni…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Robot Task Planning

Continuous Control with Coarse-to-fine Reinforcement Learning

2024-07-10 · Younggyo Seo, Jafar Uruç, Stephen James

Despite recent advances in improving the sample-efficiency of reinforcement learning (RL) algorithms, designing an RL algorithm that can be practically deployed in real-world environments remains a challenge. In this pap…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Self-Imitation Learning from Demonstrations

2022-03-21 · Georgiy Pshikhachev, Dmitry Ivanov, Vladimir Egorov, Aleksei Shpilman

Despite the numerous breakthroughs achieved with Reinforcement Learning (RL), solving environments with sparse rewards remains a challenging task that requires sophisticated exploration. Learning from Demonstrations (LfD…

Imitation Learning Reinforcement Learning (RL)