Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning

2024-06-29 · Gautham Vasan, Yan Wang, Fahim Shahriar, James Bergstra, Martin Jagersand, A. Rupam Mahmood

Many real-world robot learning problems, such as pick-and-place or arriving at a destination, can be seen as a problem of reaching a goal state as soon as possible. These problems, when formulated as episodic reinforcement learning tasks, can easily be specified to align well with our intended goal: -1 reward every time step with termination upon reaching the goal state, called minimum-time tasks. Despite this simplicity, such formulations are often overlooked in favor of dense rewards due to their perceived difficulty and lack of informativeness. Our studies contrast the two reward paradigms, revealing that the minimum-time task specification not only facilitates learning higher-quality policies but can also surpass dense-reward-based policies on their own performance metrics. Crucially, we also identify the goal-hit rate of the initial policy as a robust early indicator for learning success in such sparse feedback settings. Finally, using four distinct real-robotic platforms, we show that it is possible to learn pixel-based policies from scratch within two to three hours using constant negative rewards.

📄 PDF Abstract BibTeX arXiv:2407.00324

Code (1)

rlai-lab/relod 공식 구현 pytorch

Tasks

Informativenessreinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Unbiased Methods for Multi-Goal Reinforcement Learning

2021-06-16 · Léonard Blier, Yann Ollivier

In multi-goal reinforcement learning (RL) settings, the reward for each goal is sparse, and located in a small neighborhood of the goal. In large dimension, the probability of reaching a reward vanishes and the agent rec…

Multi-Goal Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1

Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learning

2021-07-08 · Yuexiang Zhai, Christina Baek, Zhengyuan Zhou, Jiantao Jiao 외

Many goal-reaching reinforcement learning (RL) tasks have empirically verified that rewarding the agent on subgoals improves convergence speed and practical performance. We attempt to provide a theoretical framework to q…

Hierarchical Reinforcement Learning Q-Learning Reinforcement Learning (RL)

TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations

2024-07-11 · Junik Bae, Kwanyoung Park, Youngwoon Lee

Unsupervised goal-conditioned reinforcement learning (GCRL) is a promising paradigm for developing diverse robotic skills without external supervision. However, existing unsupervised GCRL methods often struggle to cover …

Learning to Reach Goals via Iterated Supervised Learning

2019-12-12 · ICLR 2021 1 · Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu 외

Current reinforcement learning (RL) algorithms can be brittle and difficult to use, especially when learning goal-reaching behaviors from sparse rewards. Although supervised imitation learning provides a simple and stabl…

Multi-Goal Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

2017-05-18 · Nat Dilokthanakul, Christos Kaplanis, Nick Pawlowski, Murray Shanahan

The problem of sparse rewards is one of the hardest challenges in contemporary reinforcement learning. Hierarchical reinforcement learning (HRL) tackles this problem by using a set of temporally-extended actions, or opti…

Hierarchical Reinforcement Learning Montezuma's Revenge reinforcement-learning Reinforcement Learning+1