Unifying Model-Free Efficiency and Model-Based Representations via Latent Dynamics
We present Unified Latent Dynamics (ULD), a novel reinforcement learning algorithm that unifies the efficiency of model-free methods with the representational strengths of model-based approaches, without incurring planning overhead. By embedding state-action pairs into a latent space in which the true value function is approximately linear, our method supports a single set of hyperparameters across diverse domains -- from continuous control with low-dimensional and pixel inputs to high-dimensional Atari games. We prove that, under mild conditions, the fixed point of our embedding-based temporal-difference updates coincides with that of a corresponding linear model-based value expansion, and we derive explicit error bounds relating embedding fidelity to value approximation quality. In practice, ULD employs synchronized updates of encoder, value, and policy networks, auxiliary losses for short-horizon predictive dynamics, and reward-scale normalization to ensure stable learning under sparse rewards. Evaluated on 80 environments spanning Gym locomotion, DeepMind Control (proprioceptive and visual), and Atari, our approach matches or exceeds the performance of specialized model-free and general model-based baselines -- achieving cross-domain competence with minimal tuning and a fraction of the parameter footprint. These results indicate that value-aligned latent representations alone can deliver the adaptability and sample efficiency traditionally attributed to full model-based planning.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningContinuous ControlAtari GamesSimilar Papers 제목 키워드 기반
Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control
Sample-efficient policy learning from pixels is a long-standing challenge in reinforcement learning (RL). Recent dynamics-based representation learning methods have significantly improved the sample efficiency of model-f…
Representation LearningReinforcement LearningContinuous ControlStochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
Deep reinforcement learning (RL) algorithms can use high-capacity deep networks to learn directly from image observations. However, these high-dimensional observation spaces present a number of challenges in practice, si…
continuous-controlContinuous ControlDeep Reinforcement Learningreinforcement-learning+3SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels
Learning a latent dynamics model provides a task-agnostic representation of an agent's understanding of its environment. Leveraging this knowledge for model-based reinforcement learning (RL) holds the potential to improv…
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Understanding Self-Supervised Learning via Latent Distribution Matching
Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of ne…
Self-Supervised LearningConstrained latent state modeling: A unifying perspective on representation learning under competing constraints
Learning latent representations from complex data is central to modern machine learning, spanning temporal, multimodal, and partially observed systems. In such settings, representations are more naturally understood as l…
Representation Learning