Offline Reinforcement Learning with Universal Horizon Models

2026-05-15 · Hojun Chung, Junseo Lee, Songhwai Oh arxiv

Model-based reinforcement learning (RL) offers a compelling approach to offline RL by enabling value learning on imagined on-policy trajectories. However, it often suffers from compounding errors due to repeated model inference on self-generated states. While geometric horizon models (GHM) alleviate this issue through direct prediction over a discounted infinite-horizon future, they remain challenged in accurately modeling distant future states. To this end, we introduce universal horizon models (UHM), a generalization of GHM that directly predicts future states under arbitrary horizons. Leveraging this flexibility, we propose a scalable value learning method that employs a winsorized horizon distribution to stabilize training by capping excessively large horizons. Experimental results on 100 challenging OGBench tasks demonstrate that the proposed method outperforms competitive baselines, particularly on tasks with highly suboptimal datasets and those requiring long-horizon reasoning. Project page: https://rllab-snu.github.io/projects/UHM/

📄 PDF Abstract BibTeX arXiv:2605.15603

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Offline RL

Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism

2025-12-04 · Tianwei Ni, Esther Derman, Vineet Jain, Vincent Taboga 외 arxiv

Popular offline reinforcement learning (RL) methods rely on explicit conservatism, penalizing out-of-dataset actions or restricting rollout horizons. We question the universality of this principle and revisit a complemen…

Reinforcement Learning Test-time Adaptation

Test-time Offline Reinforcement Learning on Goal-related Experience

2025-07-24 · Marco Bagatella, Mert Albaba, Jonas Hübotter, Georg Martius 외 arxiv

Foundation models compress a large amount of information in a single, large neural network, which can then be queried for individual tasks. There are strong parallels between this widespread framework and offline goal-co…

Reinforcement Learning

Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learning

2024-02-15 · Yinglun Xu, Rohan Gumaste, Gagandeep Singh

We study the problem of universal black-boxed reward poisoning attacks against general offline reinforcement learning with deep neural networks. We consider a black-box threat model where the attacker is entirely oblivio…

Offline RL reinforcement-learning Reinforcement Learning

Latent Plans for Task-Agnostic Offline Reinforcement Learning

2022-09-19 · Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker 외

Everyday tasks of long-horizon and comprising a sequence of multiple implicit subtasks still impose a major challenge in offline robot control. While a number of prior methods aimed to address this setting with variants …

Imitation Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Horizon Reduction as Information Loss in Offline Reinforcement Learning

2025-12-25 · Uday Kumar Nidadala, Venkata Bhumika Guthi arxiv

Horizon reduction is a common design strategy in offline reinforcement learning (RL), used to mitigate long-horizon credit assignment, improve stability, and enable scalable learning through truncated rollouts, windowed …