Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing goal reaching performance together with the entropy of the goal distribution, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. We prove that, under regularity conditions, Skew-Fit converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that combining Skew-Fit for learning goal distributions with existing goal-reaching methods outperforms a variety of prior methods on open-sourced visual goal-reaching tasks. Moreover, we demonstrate that Skew-Fit enables a real-world robot to learn to open a door, entirely from scratch, from pixels, and without any manually-designed reward function.
Code (2)
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)validSimilar Papers 제목 키워드 기반
Log-normality and Skewness of Estimated State/Action Values in Reinforcement Learning
Under/overestimation of state/action values are harmful for reinforcement learning agents. In this paper, we show that a state/action value estimated using the Bellman equation can be decomposed to a weighted sum of path…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Scope Loss for Imbalanced Classification and RL Exploration
We demonstrate equivalence between the reinforcement learning problem and the supervised classification problem. We consequently equate the exploration exploitation trade-off in reinforcement learning to the dataset imba…
Classificationimbalanced classificationreinforcement-learningReinforcement LearningOn mechanisms for transfer using landmark value functions in multi-task lifelong reinforcement learning
Transfer learning across different reinforcement learning (RL) tasks is becoming an increasingly valuable area of research. We consider a goal-based multi-task RL framework and mechanisms by which previously solved tasks…
Reinforcement LearningReinforcement Learning (RL)Transfer LearningSymmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning
In deep reinforcement learning, estimating the value function to evaluate the quality of states and actions is essential. The value function is often trained using the least squares method, which implicitly assumes a Gau…
continuous-controlContinuous ControlDeep Reinforcement LearningMuJoCo+3Self-Supervised Discovering of Interpretable Features for Reinforcement Learning
Deep reinforcement learning (RL) has recently led to many breakthroughs on a range of complex control tasks. However, the agent's decision-making process is generally not transparent. The lack of interpretability hinders…
Atari GamesDecision MakingDeep Reinforcement Learningreinforcement-learning+2