Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning
Substantial advancements to model-based reinforcement learning algorithms have been impeded by the model-bias induced by the collected data, which generally hurts performance. Meanwhile, their inherent sample efficiency warrants utility for most robot applications, limiting potential damage to the robot and its environment during training. Inspired by information theoretic model predictive control and advances in deep reinforcement learning, we introduce Model Predictive Actor-Critic (MoPAC), a hybrid model-based/model-free method that combines model predictive rollouts with policy optimization as to mitigate model bias. MoPAC leverages optimal trajectories to guide policy learning, but explores via its model-free method, allowing the algorithm to learn more expressive dynamics models. This combination guarantees optimal skill learning up to an approximation error and reduces necessary physical interaction with the environment, making it suitable for real-robot training. We provide extensive results showcasing how our proposed method generally outperforms current state-of-the-art and conclude by evaluating MoPAC for learning on a physical robotic hand performing valve rotation and finger gaiting--a task that requires grasping, manipulation, and then regrasping of an object.
Code (1)
Tasks
Deep Reinforcement LearningModel-based Reinforcement LearningModel Predictive Controlreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
In this paper, we present a new intrinsically motivated actor-critic algorithm for learning continuous motor skills directly from raw visual input. Our neural architecture is composed of a critic and an actor network. Bo…
continuous-controlContinuous ControlRobot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models
A core challenge for an autonomous agent acting in the real world is to adapt its repertoire of skills to cope with its noisy perception and dynamics. To scale learning of skills to long-horizon tasks, robots should be a…
Deep Reinforcement LearningResidual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics
Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. Skills are typically extracted from expert demonstrations and are embedded into a la…
Reinforcement Learning (RL)Robot Skill Generalization via Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models
A long-standing challenge for a robotic manipulation system operating in real-world scenarios is adapting and generalizing its acquired motor skills to unseen environments. We tackle this challenge employing hybrid skill…
Skill GeneralizationZero-shot GeneralizationPI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale
The predictive information, the mutual information between the past and future, has been shown to be a useful representation learning auxiliary loss for training reinforcement learning agents, as the ability to model wha…
reinforcement-learningReinforcement Learning (RL)Representation LearningRobot Manipulation