Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling

2022-12-16 · Ashish Kumar, Ilya Kuzovkin

Recent advances in batch (offline) reinforcement learning have shown promising results in learning from available offline data and proved offline reinforcement learning to be an essential toolkit in learning control policies in a model-free setting. An offline reinforcement learning algorithm applied to a dataset collected by a suboptimal non-learning-based algorithm can result in a policy that outperforms the behavior agent used to collect the data. Such a scenario is frequent in robotics, where existing automation is collecting operational data. Although offline learning techniques can learn from data generated by a sub-optimal behavior agent, there is still an opportunity to improve the sample complexity of existing offline reinforcement learning algorithms by strategically introducing human demonstration data into the training process. To this end, we propose a novel approach that uses uncertainty estimation to trigger the injection of human demonstration data and guide policy training towards optimal behavior while reducing overall sample complexity. Our experiments show that this approach is more sample efficient when compared to a naive way of combining expert data with data collected from a sub-optimal agent. We augmented an existing offline reinforcement learning algorithm Conservative Q-Learning with our approach and performed experiments on data collected from MuJoCo and OffWorld Gym learning environments.

📄 PDF Abstract BibTeX arXiv:2212.08232

Code (0)

등록된 구현이 없습니다.

Tasks

MuJoCoQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

2023-06-13 · Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang 외

Offline reinforcement learning (RL) provides a promising solution to learning an agent fully relying on a data-driven paradigm. However, constrained by the limited quality of the offline dataset, its performance is often…

D4RL Efficient Exploration Offline RL reinforcement-learning+1

Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning

2023-10-27 · Nicholas E. Corrado, Yuxiao Qu, John U. Balis, Adam Labiosa 외

In offline reinforcement learning (RL), an RL agent learns to solve a task using only a fixed dataset of previously collected data. While offline RL has been successful in learning real-world robot control policies, it t…

Autonomous Driving D4RL Data Augmentation Imitation Learning+4

BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models

2026-05-28 · Zhongxi Chen, Yifan Han, Yanming Shao, Huanming Liu 외 arxiv

Vision-Language-Action (VLA) policies provide strong behavioral priors for dexterous manipulation, yet adapting them on real robots remains challenging because high-DoF contact failures are difficult for humans to correc…

Reinforcement Learning

Offline Robotic World Model: Learning Robotic Policies without a Physics Simulator

2025-04-23 · Chenhao Li, Andreas Krause, Marco Hutter

Reinforcement Learning (RL) has demonstrated impressive capabilities in robotic control but remains challenging due to high sample complexity, safety concerns, and the sim-to-real gap. While offline RL eliminates the nee…

Offline RL Reinforcement Learning (RL)

An offline approach to fNIRS-guided reinforcement learning for robot behavior

2026-07-15 · Julia Santaniello, Madelaine Brower, Benson Jiang, Donatello Sassaroli 외 arxiv

Human-in-the-loop Reinforcement Learning has become a popular approach for training, finetuning, and aligning robot behavior with user preferences. Our paper explores the feasibility of using brain signals via functional…