Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discovery

2022-12-02 · Yiqin Yang, Hao Hu, Wenzhe Li, Siyuan Li, Jun Yang, Qianchuan Zhao, Chongjie Zhang

Offline reinforcement learning (RL) enables the agent to effectively learn from logged data, which significantly extends the applicability of RL algorithms in real-world scenarios where exploration can be expensive or unsafe. Previous works have shown that extracting primitive skills from the recurring and temporally extended structures in the logged data yields better learning. However, these methods suffer greatly when the primitives have limited representation ability to recover the original policy space, especially in offline settings. In this paper, we give a quantitative characterization of the performance of offline hierarchical learning and highlight the importance of learning lossless primitives. To this end, we propose to use a \emph{flow}-based structure as the representation for low-level policies. This allows us to represent the behaviors in the dataset faithfully while keeping the expression ability to recover the whole policy space. We show that such lossless primitives can drastically improve the performance of hierarchical policies. The experimental results and extensive ablation studies on the standard D4RL benchmark show that our method has a good representation ability for policies and achieves superior performance in most tasks.

📄 PDF Abstract BibTeX arXiv:2212.01105

Code (0)

등록된 구현이 없습니다.

Tasks

D4RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

LCM: Lossless Context Management

2026-02-14 · Clint Ehrlich, Theodore Blackman arxiv

We introduce Lossless Context Management (LCM), a deterministic architecture for LLM memory that outperforms Claude Code on long-context tasks. When benchmarked using Opus 4.6, our LCM-augmented coding agent, Volt, achie…

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning

2020-10-26 · ICLR 2021 1 · Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine 외

Reinforcement learning (RL) has achieved impressive performance in a variety of online settings in which an agent's ability to query the environment for transitions and rewards is effectively unlimited. However, in many …

Few-Shot Imitation Learning Imitation Learning Offline RL reinforcement-learning+2

Online Optimization in Games via Control Theory: Connecting Regret, Passivity and Poincaré Recurrence

2021-06-09 · Yun Kuen Cheung, Georgios Piliouras

We present a novel control-theoretic understanding of online optimization and learning in games, via the notion of passivity. Passivity is a fundamental concept in control theory, which abstracts energy conservation and …

Flow Matching for Offline Reinforcement Learning with Discrete Actions

2026-02-05 · Fairoz Nower Khan, Nabuat Zaman Nahim, Ruiquan Huang, Haibo Yang 외 arxiv

Generative policies based on diffusion models and flow matching have shown strong promise for offline reinforcement learning (RL), but their applicability remains largely confined to continuous action spaces. To address …

Reinforcement Learning Offline RL

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

2026-03-16 · Mumuksh Tayal, Manan Tayal, Ravi Prakash arxiv

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inferen…