Distributional Successor Features Enable Zero-Shot Policy Optimization

2024-03-10 · Chuning Zhu, Xinqi Wang, Tyler Han, Simon S. Du, Abhishek Gupta

Intelligent agents must be generalists, capable of quickly adapting to various tasks. In reinforcement learning (RL), model-based RL learns a dynamics model of the world, in principle enabling transfer to arbitrary reward functions through planning. However, autoregressive model rollouts suffer from compounding error, making model-based RL ineffective for long-horizon problems. Successor features offer an alternative by modeling a policy's long-term state occupancy, reducing policy evaluation under new rewards to linear regression. Yet, zero-shot policy optimization for new tasks with successor features can be challenging. This work proposes a novel class of models, i.e., Distributional Successor Features for Zero-Shot Policy Optimization (DiSPOs), that learn a distribution of successor features of a stationary dataset's behavior policy, along with a policy that acts to realize different successor features achievable within the dataset. By directly modeling long-term outcomes in the dataset, DiSPOs avoid compounding error while enabling a simple scheme for zero-shot policy optimization across reward functions. We present a practical instantiation of DiSPOs using diffusion models and show their efficacy as a new class of transferable models, both theoretically and empirically across various simulated robotics problems. Videos and code available at https://weirdlabuw.github.io/dispo/.

📄 PDF Abstract BibTeX arXiv:2403.06328

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

A Distributional Analogue to the Successor Representation

2024-02-13 · Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Yunhao Tang 외

This paper contributes a new approach for distributional reinforcement learning which elucidates a clean separation of transition structure and reward in the learning process. Analogous to how the successor representatio…

Distributional Reinforcement Learning Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning

Zero-Shot Off-Policy Learning

2026-02-02 · Arip Asadulaev, Maksim Bobrin, Salem Lahlou, Dmitry Dylov 외 arxiv

Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions. This objective presents significant challenges, primarily due to the inherent distributional shift and val…

Reinforcement Learning Continuous Control

Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning

2026-05-13 · Stefan Stojanovic, Alexandre Proutiere arxiv

Hierarchical reinforcement learning can improve generalization by decomposing long-horizon decision-making into simpler subproblems. However, existing approaches often rely on restrictive design choices, such as fixed te…

Hierarchical Reinforcement Learning

Zero-Shot Reinforcement Learning from Low Quality Data

2023-09-26 · Scott Jeen, Tom Bewley, Jonathan M. Cullen

Zero-shot reinforcement learning (RL) promises to provide agents that can perform any task in an environment after an offline, reward-free pre-training phase. Methods leveraging successor measures and successor features …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A neurally plausible model learns successor representations in partially observable environments

2019-06-22 · NeurIPS 2019 12 · Eszter Vertes, Maneesh Sahani

Animals need to devise strategies to maximize returns while interacting with their environment based on incoming noisy sensory observations. Task-relevant states, such as the agent's location within an environment or the…