Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspective

2022-09-26 · Lunjun Zhang, Bradly C. Stadie

Hindsight goal relabeling has become a foundational technique in multi-goal reinforcement learning (RL). The essential idea is that any trajectory can be seen as a sub-optimal demonstration for reaching its final state. Intuitively, learning from those arbitrary demonstrations can be seen as a form of imitation learning (IL). However, the connection between hindsight goal relabeling and imitation learning is not well understood. In this paper, we propose a novel framework to understand hindsight goal relabeling from a divergence minimization perspective. Recasting the goal reaching problem in the IL framework not only allows us to derive several existing methods from first principles, but also provides us with the tools from IL to improve goal reaching algorithms. Experimentally, we find that under hindsight relabeling, Q-learning outperforms behavioral cloning (BC). Yet, a vanilla combination of both hurts performance. Concretely, we see that the BC loss only helps when selectively applied to actions that get the agent closer to the goal according to the Q-function. Our framework also explains the puzzling phenomenon wherein a reward of (-1, 0) results in significantly better performance than a (0, 1) reward for goal reaching.

📄 PDF Abstract BibTeX arXiv:2209.13046

Code (0)

등록된 구현이 없습니다.

Tasks

Imitation LearningMulti-Goal Reinforcement LearningQ-LearningReinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Goal Sets, Not Goal States: Queryable Robot Goals through Goal-Set Hindsight Relabeling

2026-06-08 · Carlos Vélez García, Miguel Cazorla, Jorge Pomares arxiv

Hindsight relabeling usually turns achieved future states into exact goals, which can overconstrain offline robot learning when task success depends only on a subset of the state. We propose Goal-Set Hindsight Relabeling…

Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

2020-02-25 · NeurIPS 2020 12 · Benjamin Eysenbach, Xinyang Geng, Sergey Levine, Ruslan Salakhutdinov

Multi-task reinforcement learning (RL) aims to simultaneously learn policies for solving many tasks. Several prior works have found that relabeling past experience with different reward functions can improve sample effic…

Reinforcement Learning Reinforcement Learning (RL)

GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning

2025-08-08 · Xing Lei, Wenyan Yang, Kaiqiang Ke, Shentao Yang 외 arxiv

Goal-conditioned reinforcement learning (GCRL) with sparse rewards remains a fundamental challenge in reinforcement learning. While hindsight experience replay (HER) has shown promise by relabeling collected trajectories…

Reinforcement Learning

Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning

2025-05-06 · Caleb Chuck, Fan Feng, Carl Qi, Chang Shi 외

Hindsight relabeling is a powerful tool for overcoming sparsity in goal-conditioned reinforcement learning (GCRL), especially in certain domains such as navigation and locomotion. However, hindsight relabeling can strugg…

counterfactual Object reinforcement-learning Reinforcement Learning

Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning

2021-02-25 · Rui Yang, Jiafei Lyu, Yu Yang, Jiangpeng Yan 외

Multi-goal reinforcement learning is widely applied in planning and robot manipulation. Two main challenges in multi-goal reinforcement learning are sparse rewards and sample inefficiency. Hindsight Experience Replay (HE…

Multi-Goal Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1