Residual Reinforcement Learning from Demonstrations

2021-06-15 · Minttu Alakuijala, Gabriel Dulac-Arnold, Julien Mairal, Jean Ponce, Cordelia Schmid

Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal. We extend the residual formulation to learn from visual inputs and sparse rewards using demonstrations. Learning from images, proprioceptive inputs and a sparse task-completion reward relaxes the requirement of accessing full state features, such as object and target positions. In addition, replacing the base controller with a policy learned from demonstrations removes the dependency on a hand-engineered controller in favour of a dataset of demonstrations, which can be provided by non-experts. Our experimental evaluation on simulated manipulation tasks on a 6-DoF UR5 arm and a 28-DoF dexterous hand demonstrates that residual RL from demonstrations is able to generalize to unseen environment conditions more flexibly than either behavioral cloning or RL fine-tuning, and is capable of solving high-dimensional, sparse-reward tasks out of reach for RL from scratch.

📄 PDF Abstract BibTeX arXiv:2106.08050

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Difference of Convex Functions Programming Applied to Control with Expert Data

2016-06-03 · Bilal Piot, Matthieu Geist, Olivier Pietquin

This paper reports applications of Difference of Convex functions (DC) programming to Learning from Demonstrations (LfD) and Reinforcement Learning (RL) with expert data. This is made possible because the norm of the Opt…

General Classification reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approach

2026-05-27 · Guang-Yuan Hao, Lars van der Laan, Aurélien Bibaut, Nathan Kallus arxiv

We study the transfer of rewards learned using inverse reinforcement learning from expert demonstrations in one environment to reinforcement learning in a new, different environment. This arises naturally when demonstrat…

Reinforcement Learning

Self-Improving Vision-Language-Action Models with Data Generation via Residual RL

2025-10-30 · Wenli Xiao, Haotian Lin, Andy Peng, Haoru Xue 외 arxiv

Supervised fine-tuning (SFT) has become the de facto post-training strategy for large vision-language-action (VLA) models, but its reliance on costly human demonstrations limits scalability and generalization. We propose…

Reinforcement Learning

DexFlyWheel: A Scalable and Self-improving Data Generation Framework for Dexterous Manipulation

2025-09-28 · Kefei Zhu, Fengshuo Bai, YuanHao Xiang, Yishuai Cai 외 arxiv

Dexterous manipulation is critical for advancing robot capabilities in real-world applications, yet diverse and high-quality datasets remain scarce. Existing data collection methods either rely on human teleoperation or …

Reinforcement Learning Data Augmentation

Inverse Manipulation through Symbolic Planning and Residual Operator Learning

2026-06-03 · Yigit Yildirim, Giuseppe Rauso, Riccardo Caccavale, Alberto Finzi arxiv

Inverting a robotic task requires more than reversing symbolic state transitions or rewinding motor trajectories. In robot manipulation tasks, symbolic inverse plans often fail to fully restore the effects of forward exe…

Reinforcement Learning Robot Manipulation