Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning

2023-07-07 · NeurIPS 2023 11 · Seungyong Moon, Junyoung Yeom, Bumsoo Park, Hyun Oh Song

Discovering achievements with a hierarchical structure in procedurally generated environments presents a significant challenge. This requires an agent to possess a broad range of abilities, including generalization and long-term reasoning. Many prior methods have been built upon model-based or hierarchical approaches, with the belief that an explicit module for long-term planning would be advantageous for learning hierarchical dependencies. However, these methods demand an excessive number of environment interactions or large model sizes, limiting their practicality. In this work, we demonstrate that proximal policy optimization (PPO), a simple yet versatile model-free algorithm, outperforms previous methods when optimized with recent implementation practices. Moreover, we find that the PPO agent can predict the next achievement to be unlocked to some extent, albeit with limited confidence. Based on this observation, we introduce a novel contrastive learning method, called achievement distillation, which strengthens the agent's ability to predict the next achievement. Our method exhibits a strong capacity for discovering hierarchical achievements and shows state-of-the-art performance on the challenging Crafter environment in a sample-efficient manner while utilizing fewer model parameters.

📄 PDF Abstract BibTeX arXiv:2307.03486

Code (1)

snu-mllab/Achievement-Distillation 공식 구현 pytorch

Tasks

Contrastive Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Contrastive Learning 설명 없음

Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL

2026-05-25 · Sarthak Dayal, Abhinav Peri, Carl Qi, Claas Voelcker 외 arxiv

Hierarchical Reinforcement Learning (HRL) promises to solve long-horizon Reinforcement Learning (RL) tasks more efficiently than non-hierarchical counterparts by discovering and reusing temporally-extended skills. Howeve…

Hierarchical Reinforcement Learning

DisTop: Discovering a Topological representation to learn diverse and rewarding skills

2021-06-06 · Arthur Aubret, Laetitia Matignon, Salima Hassas

The optimal way for a deep reinforcement learning (DRL) agent to explore is to learn a set of skills that achieves a uniform distribution of states. Following this,we introduce DisTop, a new model that simultaneously lea…

Deep Reinforcement Learning Hierarchical Reinforcement Learning MuJoCo reinforcement-learning+3

Incremental Hierarchical Reinforcement Learning with Multitask LMDPs

2018-09-27 · Adam C Earle, Andrew M Saxe, Benjamin Rosman

Exploration is a well known challenge in Reinforcement Learning. One principled way of overcoming this challenge is to find a hierarchical abstraction of the base problem and explore at these higher levels, rather than i…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Fine-grained Category Discovery under Coarse-grained supervision with Hierarchical Weighted Self-contrastive Learning

2022-10-14 · Wenbin An, Feng Tian, Ping Chen, Siliang Tang 외

Novel category discovery aims at adapting models trained on known categories to novel categories. Previous works only focus on the scenario where known and novel categories are of the same granularity. In this paper, we …

Contrastive Learning

Hierarchical Reinforcement Learning By Discovering Intrinsic Options

2021-01-16 · ICLR 2021 1 · Jesse Zhang, Haonan Yu, Wei Xu

We propose a hierarchical reinforcement learning method, HIDIO, that can learn task-agnostic options in a self-supervised manner while jointly learning to utilize them to solve sparse-reward tasks. Unlike current hierarc…