Off-Policy Deep Reinforcement Learning with Analogous Disentangled Exploration
Off-policy reinforcement learning (RL) is concerned with learning a rewarding policy by executing another policy that gathers samples of experience. While the former policy (i.e. target policy) is rewarding but in-expressive (in most cases, deterministic), doing well in the latter task, in contrast, requires an expressive policy (i.e. behavior policy) that offers guided and effective exploration. Contrary to most methods that make a trade-off between optimality and expressiveness, disentangled frameworks explicitly decouple the two objectives, which each is dealt with by a distinct separate policy. Although being able to freely design and optimize the two policies with respect to their own objectives, naively disentangling them can lead to inefficient learning or stability issues. To mitigate this problem, our proposed method Analogous Disentangled Actor-Critic (ADAC) designs analogous pairs of actors and critics. Specifically, ADAC leverages a key property about Stein variational gradient descent (SVGD) to constraint the expressive energy-based behavior policy with respect to the target one for effective exploration. Additionally, an analogous critic pair is introduced to incorporate intrinsic rewards in a principled manner, with theoretical guarantees on the overall learning stability and effectiveness. We empirically evaluate environment-reward-only ADAC on 14 continuous-control tasks and report the state-of-the-art on 10 of them. We further demonstrate ADAC, when paired with intrinsic rewards, outperform alternatives in exploration-challenging tasks.
Code (1)
Tasks
continuous-controlContinuous ControlDeep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Zero-Shot Policy Transfer with Disentangled Task Representation of Meta-Reinforcement Learning
Humans are capable of abstracting various tasks as different combinations of multiple attributes. This perspective of compositionality is vital for human rapid learning and adaption since previous experiences from relate…
DisentanglementMeta Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Entropy Augmented Reinforcement Learning
Deep reinforcement learning was instigated with the presence of trust region methods, being scalable and efficient. However, the pessimism of such algorithms, among which it forces to constrain in a trust region by all m…
Deep Reinforcement LearningMuJoCoreinforcement-learningReinforcement Learning+1Provably Safe PAC-MDP Exploration Using Analogies
A key challenge in applying reinforcement learning to safety-critical domains is understanding how to balance exploration (needed to attain good performance on the task) with safety (needed to avoid catastrophic failure)…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe ExplorationWeakly-Supervised Learning of Disentangled and Interpretable Skills for Hierarchical Reinforcement Learning
Hierarchical reinforcement learning (RL) usually requires task-agnostic and interpretable skills that can be applicable to various downstream tasks. While many recent works have been proposed to learn such skills for a p…
DecoderHierarchical Reinforcement LearningInductive Biasreinforcement-learning+2Model Learning for Look-ahead Exploration in Continuous Control
We propose an exploration method that incorporates look-ahead search over basic learnt skills and their dynamics, and use it for reinforcement learning (RL) of manipulation policies . Our skills are multi-goal policies l…
continuous-controlContinuous ControlReinforcement LearningReinforcement Learning (RL)