Batch Exploration with Examples for Scalable Robotic Reinforcement Learning
Learning from diverse offline datasets is a promising path towards learning general purpose robotic agents. However, a core challenge in this paradigm lies in collecting large amounts of meaningful data, while not depending on a human in the loop for data collection. One way to address this challenge is through task-agnostic exploration, where an agent attempts to explore without a task-specific reward function, and collect data that can be useful for any downstream task. While these approaches have shown some promise in simple domains, they often struggle to explore the relevant regions of the state space in more challenging settings, such as vision based robotic manipulation. This challenge stems from an objective that encourages exploring everything in a potentially vast state space. To mitigate this challenge, we propose to focus exploration on the important parts of the state space using weak human supervision. Concretely, we propose an exploration technique, Batch Exploration with Examples (BEE), that explores relevant regions of the state-space, guided by a modest number of human provided images of important states. These human provided images only need to be collected once at the beginning of data collection and can be collected in a matter of minutes, allowing us to scalably collect diverse datasets, which can then be combined with any batch RL algorithm. We find that BEE is able to tackle challenging vision-based manipulation tasks both in simulation and on a real Franka robot, and observe that compared to task-agnostic and weakly-supervised exploration techniques, it (1) interacts more than twice as often with relevant objects, and (2) improves downstream task performance when used in conjunction with offline RL.
Code (1)
Tasks
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
Recent high-capacity vision-language-action (VLA) models have demonstrated impressive performance on a range of robotic manipulation tasks by imitating human demonstrations. However, exploiting offline data with limited …
GPUReinforcement Learning (RL)Vision-Language-ActionEvaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks
Deep reinforcement learning has been shown to solve challenging tasks where large amounts of training experience is available, usually obtained online while learning the task. Robotics is a significant potential applicat…
continuous-controlContinuous ControlDeep Reinforcement LearningReinforcement LearningMT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
General-purpose robotic systems must master a large repertoire of diverse skills to be useful in a range of daily tasks. While reinforcement learning provides a powerful framework for acquiring individual behaviors, the …
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Online inductive learning from answer sets for efficient reinforcement learning exploration
This paper presents a novel approach combining inductive logic programming with reinforcement learning to improve training performance and explainability. We exploit inductive learning of answer set programs from noisy e…
Inductive LearningInductive logic programmingQ-Learningreinforcement-learning+1Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow
Controlling high-dimensional systems in biological and robotic applications is challenging due to expansive state-action spaces, where effective exploration is critical. Commonly used exploration strategies in reinforcem…
Dimensionality ReductionReinforcement LearningContinuous Control