Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning

2018-05-29 · ICLR 2018 1 · Wen Sun, J. Andrew Bagnell, Byron Boots

In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the near-optimal cost-to-go oracle on the planning horizon and demonstrate that the cost-to-go oracle shortens the learner's planning horizon as function of its accuracy: a globally optimal oracle can shorten the planning horizon to one, leading to a one-step greedy Markov Decision Process which is much easier to optimize, while an oracle that is far away from the optimality requires planning over a longer horizon to achieve near-optimal performance. Hence our new insight bridges the gap and interpolates between imitation learning and reinforcement learning. Motivated by the above mentioned insights, we propose Truncated HORizon Policy Search (THOR), a method that focuses on searching for policies that maximize the total reshaped reward over a finite planning horizon when the oracle is sub-optimal. We experimentally demonstrate that a gradient-based implementation of THOR can achieve superior performance compared to RL baselines and IL baselines even when the oracle is sub-optimal.

📄 PDF Abstract BibTeX arXiv:1805.11240

Code (0)

등록된 구현이 없습니다.

Tasks

Imitation Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Are Full Rollouts Necessary for On-Policy Distillation?

2026-05-29 · Yaocheng Zhang, Jiajun Chai, Yuqian Fu, Songjun Tu 외 arxiv

On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts rather than fixed teacher traces and has emerged as a promising post-training paradigm. However, standard OPD typically genera…

Reinforcement Learning Mathematical Reasoning

Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling

2026-04-10 · Xubin Zhou, Yipeng Yang, Zhan Li arxiv

Maximum entropy reinforcement learning (MaxEnt RL) has become a standard framework for sequential decision making, yet its standard Gaussian policy parameterization is inherently unimodal, limiting its ability to model c…

Reinforcement Learning Decision Making

Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation

2025-10-29 · Feichen Gan, Youcun Lu, Yingying Zhang, Yukun Liu arxiv

Reliable uncertainty quantification is crucial for reinforcement learning (RL) in high-stakes settings. We propose a unified conformal prediction framework for infinite-horizon policy evaluation that constructs distribut…

Reinforcement Learning

Nearly Minimax Optimal Reward-free Reinforcement Learning

2020-10-12 · Zihan Zhang, Simon S. Du, Xiangyang Ji

We study the reward-free reinforcement learning framework, which is particularly suitable for batch reinforcement learning and scenarios where one needs policies for multiple reward functions. This framework has two phas…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Improving Stochastic Action-Constrained Reinforcement Learning via Truncated Distributions

2025-11-27 · Roland Stolz, Michael Eichelbeck, Matthias Althoff arxiv

In reinforcement learning (RL), it is often advantageous to consider additional constraints on the action space to ensure safety or action relevance. Existing work on such action-constrained RL faces challenges regarding…

Computational Efficiency Reinforcement Learning