Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information

2021-10-21 · Jin Li, Xianyuan Zhan, Zixu Xiao, Guyue Zhou

End-to-end learning robotic manipulation with high data efficiency is one of the key challenges in robotics. The latest methods that utilize human demonstration data and unsupervised representation learning has proven to be a promising direction to improve RL learning efficiency. The use of demonstration data also allows "warming-up" the RL policies using offline data with imitation learning or the recently emerged offline reinforcement learning algorithms. However, existing works often treat offline policy learning and online exploration as two separate processes, which are often accompanied by severe performance drop during the offline-to-online transition. Furthermore, many robotic manipulation tasks involve complex sub-task structures, which are very challenging to be solved in RL with sparse reward. In this work, we propose a unified offline-to-online RL framework that resolves the transition performance drop issue. Additionally, we introduce goal-aware state information to the RL agent, which can greatly reduce task complexity and accelerate policy learning. Combined with an advanced unsupervised representation learning module, our framework achieves great training efficiency and performance compared with the state-of-the-art methods in multiple robotic manipulation tasks.

📄 PDF Abstract BibTeX arXiv:2110.10905

Code (0)

등록된 구현이 없습니다.

Tasks

Imitation LearningReinforcement Learning (RL)Representation Learning

Benchmarking Offline Reinforcement Learning on Real-Robot Hardware

2023-07-28 · Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier 외

Learning policies from previously recorded data is a promising direction for real-world robotics tasks, as online learning is often infeasible. Dexterous manipulation in particular remains an open problem in its general …

Benchmarking reinforcement-learning Reinforcement Learning

Equivariant Offline Reinforcement Learning

2024-06-20 · Arsh Tangri, Ondrej Biza, Dian Wang, David Klee 외

Sample efficiency is critical when applying learning-based methods to robotic manipulation due to the high cost of collecting expert demonstrations and the challenges of on-robot policy learning through online Reinforcem…

Offline RL Q-Learning reinforcement-learning Reinforcement Learning+1

Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble

2021-07-01 · SeungHyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel 외

Recent advance in deep offline reinforcement learning (RL) has made it possible to train strong robotic agents from offline datasets. However, depending on the quality of the trained agents and the application being cons…

Offline RL reinforcement-learning Reinforcement Learning (RL)

How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation

2022-05-06 · Alex X. Lee, Coline Devin, Jost Tobias Springenberg, Yuxiang Zhou 외

Reinforcement learning (RL) has been shown to be effective at learning control from experience. However, RL typically requires a large amount of online interaction with the environment. This limits its applicability to r…

Offline RL Reinforcement Learning (RL)

Efficient Reinforcement Learning by Guiding Generalist World Models with Non-Curated Data

2025-02-26 · Yi Zhao, Aidan Scannell, Wenshuai Zhao, Yuxin Hou 외

Leveraging offline data is a promising way to improve the sample efficiency of online reinforcement learning (RL). This paper expands the pool of usable data for offline-to-online RL by leveraging abundant non-curated da…

Attribute reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)