Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks

2018-10-06 · ICLR 2019 5 · Fabio Pardo, Vitaly Levdik, Petar Kormushev

Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in epsilon-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.

📄 PDF Abstract BibTeX arXiv:1810.02927

Code (2)

fabiopardo/qmap tf

yl3829/Q-map tf

Tasks

AllMontezuma's RevengeNavigateQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)SNES GamesSokoban

Recursive Deep Inverse Reinforcement Learning

2025-04-17 · Paul Ghanem, Michael Potter, Owen Howell, Pau Closas 외

Inferring an adversary's goals from exhibited behavior is crucial for counterplanning and non-cooperative multi-agent systems in domains like cybersecurity, military, and strategy games. Deep Inverse Reinforcement Learni…

reinforcement-learning Reinforcement Learning

Many-Goals Reinforcement Learning

2018-06-22 · Vivek Veeriah, Junhyuk Oh, Satinder Singh

All-goals updating exploits the off-policy nature of Q-learning to update all possible goals an agent could have from each transition in the world, and was introduced into Reinforcement Learning (RL) by Kaelbling (1993).…

All Q-Learning reinforcement-learning Reinforcement Learning+1

Vision-Language Models as a Source of Rewards

2023-12-14 · Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan 외

Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for building generalist agents with RL has been…

reinforcement-learning Reinforcement Learning

Goal-Conditioned Agents that Learn Everything All at Once

2026-05-22 · Michael Matthews, Matthew Jackson, Michael Beukman, Thomas Foster 외 arxiv

A goal-conditioned reinforcement learning agent exploring an environment will see a wealth of information throughout a trajectory, most of which is discarded when only performing on-policy updates with respect to the com…

Reinforcement Learning Continuous Control

ViSA: Visited-State Augmentation for Generalized Goal-Space Contrastive Reinforcement Learning

2026-03-16 · Issa Nakamura, Tomoya Yamanokuchi, Yuki Kadokawa, Jia Qu 외 arxiv

Goal-Conditioned Reinforcement Learning (GCRL) is a framework for learning a policy that can reach arbitrarily given goals. In particular, Contrastive Reinforcement Learning (CRL) provides a framework for policy updates …

Reinforcement Learning Contrastive Learning Data Augmentation