Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
Using a model of the environment, reinforcement learning agents can plan their future moves and achieve superhuman performance in board games like Chess, Shogi, and Go, while remaining relatively sample-efficient. As demonstrated by the MuZero Algorithm, the environment model can even be learned dynamically, generalizing the agent to many more tasks while at the same time achieving state-of-the-art performance. Notably, MuZero uses internal state representations derived from real environment states for its predictions. In this paper, we bind the model's predicted internal state representation to the environment state via two additional terms: a reconstruction model loss and a simpler consistency loss, both of which work independently and unsupervised, acting as constraints to stabilize the learning process. Our experiments show that this new integration of reconstruction model loss and simpler consistency loss provide a significant performance increase in OpenAI Gym environments. Our modifications also enable self-supervised pretraining for MuZero, so the algorithm can learn about environment dynamics before a goal is made available.
Code (2)
Tasks
Board GamesModel-based Reinforcement LearningOpenAI Gymreinforcement-learningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Visual Reinforcement Learning with Self-Supervised 3D Representations
A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved sample-efficiency and generalization thro…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Representation Learning+1OISD: On-Policy Internal Self-Distillation of Language Models
Recent reinforcement learning (RL) post-training approaches primarily optimize the final output policy using sparse outcome-level rewards, while largely overlooking predictive signals encoded in intermediate representati…
Reinforcement LearningMathematical ReasoningVQEL: Enabling Self-Developed Symbolic Language in Agents through Vector Quantization in Emergent Language Games
In the field of emergent language, efforts have traditionally focused on developing communication protocols through interactions between agents in referential games. However, the aspect of internal language learning, whe…
QuantizationExploration Through Introspection: A Self-Aware Reward Model
Understanding how artificial agents model internal mental states is central to advancing Theory of Mind in AI. Evidence points to a unified system for self- and other-awareness. We explore this self-awareness by having r…
Reinforcement LearningLearning sparse representations in reinforcement learning
Reinforcement learning (RL) algorithms allow artificial agents to improve their selection of actions to increase rewarding experiences in their environments. Temporal Difference (TD) Learning -- a model-free RL method --…
Acrobotreinforcement-learningReinforcement LearningReinforcement Learning (RL)