Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning
Meta-reinforcement learning enables artificial agents to learn from related training tasks and adapt to new tasks efficiently with minimal interaction data. However, most existing research is still limited to narrow task distributions that are parametric and stationary, and does not consider out-of-distribution tasks during the evaluation, thus, restricting its application. In this paper, we propose MoSS, a context-based Meta-reinforcement learning algorithm based on Self-Supervised task representation learning to address this challenge. We extend meta-RL to broad non-parametric task distributions which have never been explored before, and also achieve state-of-the-art results in non-stationary and out-of-distribution tasks. Specifically, MoSS consists of a task inference module and a policy module. We utilize the Gaussian mixture model for task representation to imitate the parametric and non-parametric task variations. Additionally, our online adaptation strategy enables the agent to react at the first sight of a task change, thus being applicable in non-stationary tasks. MoSS also exhibits strong generalization robustness in out-of-distributions tasks which benefits from the reliable and robust task representation. The policy is built on top of an off-policy RL algorithm and the entire network is trained completely off-policy to ensure high sample efficiency. On MuJoCo and Meta-World benchmarks, MoSS outperforms prior works in terms of asymptotic performance, sample efficiency (3-50x faster), adaptation efficiency, and generalization robustness on broad and diverse task distributions.
Code (0)
등록된 구현이 없습니다.
Tasks
Meta Reinforcement LearningMuJoCoreinforcement-learningReinforcement LearningRepresentation LearningSimilar Papers 제목 키워드 기반
Contextual Latent World Models for Offline Meta Reinforcement Learning
Offline meta-reinforcement learning seeks to learn policies that generalize across related tasks from fixed datasets. Context-based methods infer a task representation from transition histories, but learning effective ta…
Representation LearningReinforcement LearningImproving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
Meta-reinforcement learning typically requires orders of magnitude more samples than single task reinforcement learning methods. This is because meta-training needs to deal with more diverse distributions and train extra…
Contrastive LearningMeta Reinforcement LearningMuJoCoreinforcement-learning+3Unsupervised Meta-Testing with Conditional Neural Processes for Hybrid Meta-Reinforcement Learning
We introduce Unsupervised Meta-Testing with Conditional Neural Processes (UMCNP), a novel hybrid few-shot meta-reinforcement learning (meta-RL) method that uniquely combines, yet distinctly separates, parameterized polic…
continuous-controlContinuous ControlMeta Reinforcement LearningTowards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning
As a marriage between offline RL and meta-RL, the advent of offline meta-reinforcement learning (OMRL) has shown great promise in enabling RL agents to multi-task and quickly adapt while acquiring knowledge safely. Among…
Meta Reinforcement LearningOffline RLreinforcement-learningReinforcement Learning+1Disentangled Predictive Representation for Meta-Reinforcement Learning
A major challenge in reinforcement learning is the design of agents that are able to generalize across tasks that share common dynamics. A viable solution is meta-reinforcement learning, which identifies common struc…
Meta Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)