Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
Efficient exploration remains a challenging problem in reinforcement learning, especially for tasks where extrinsic rewards from environments are sparse or even totally disregarded. Significant advances based on intrinsic motivation show promising results in simple environments but often get stuck in environments with multimodal and stochastic dynamics. In this work, we propose a variational dynamic model based on the conditional variational inference to model the multimodality and stochasticity. We consider the environmental state-action transition as a conditional generative process by generating the next-state prediction under the condition of the current state, action, and latent variable, which provides a better understanding of the dynamics and leads a better performance in exploration. We derive an upper bound of the negative log-likelihood of the environmental transition and use such an upper bound as the intrinsic reward for exploration, which allows the agent to learn skills by self-supervised exploration without observing extrinsic rewards. We evaluate the proposed method on several image-based simulation tasks and a real robotic manipulating task. Our method outperforms several state-of-the-art environment model-based exploration approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningEfficient Explorationreinforcement-learningReinforcement LearningReinforcement Learning (RL)Variational InferenceSimilar Papers 제목 키워드 기반
TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis
Despite significant advances in Large Reasoning Models (LRMs) driven by reinforcement learning with verifiable rewards (RLVR), this paradigm is fundamentally limited in specialized or novel domains where such supervision…
Reinforcement LearningTest-time AdaptationPhysically-Grounded Goal Imagination: Physics-Informed Variational Autoencoder for Self-Supervised Reinforcement Learning
Self-supervised goal-conditioned reinforcement learning enables robots to autonomously acquire diverse skills without human supervision. However, a central challenge is the goal setting problem: robots must propose feasi…
Reinforcement LearningVASE: Variational Assorted Surprise Exploration for Reinforcement Learning
Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). Recently, surprise has been used as an intrinsic reward that encourages systematic and effici…
continuous-controlContinuous ControlEfficient Explorationreinforcement-learning+3Variational Policy Search via Trajectory Optimization
In order to learn effective control policies for dynamical systems, policy search methods must be able to discover successful executions of the desired task. While random exploration can work well in simple domains, comp…
Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL
In this work, we take a first step toward elucidating the mechanisms behind emergent exploration in unsupervised reinforcement learning. We study Single-Goal Contrastive Reinforcement Learning (SGCRL), a self-supervised …
Reinforcement Learning