Contrastive Abstraction for Reinforcement Learning

2024-10-01 · Vihang Patil, Markus Hofmarcher, Elisabeth Rumetshofer, Sepp Hochreiter

Learning agents with reinforcement learning is difficult when dealing with long trajectories that involve a large number of states. To address these learning problems effectively, the number of states can be reduced by abstract representations that cluster states. In principle, deep reinforcement learning can find abstract states, but end-to-end learning is unstable. We propose contrastive abstraction learning to find abstract states, where we assume that successive states in a trajectory belong to the same abstract state. Such abstract states may be basic locations, achieved subgoals, inventory, or health conditions. Contrastive abstraction learning first constructs clusters of state representations by contrastive learning and then applies modern Hopfield networks to determine the abstract states. The first phase of contrastive abstraction learning is self-supervised learning, where contrastive learning forces states with sequential proximity to have similar representations. The second phase uses modern Hopfield networks to map similar state representations to the same fixed point, i.e.\ to an abstract state. The level of abstraction can be adjusted by determining the number of fixed points of the modern Hopfield network. Furthermore, \textit{contrastive abstraction learning} does not require rewards and facilitates efficient reinforcement learning for a wide range of downstream tasks. Our experiments demonstrate the effectiveness of contrastive abstraction learning for reinforcement learning.

📄 PDF Abstract BibTeX arXiv:2410.00704

Code (0)

등록된 구현이 없습니다.

Tasks

Contrastive LearningDeep Reinforcement Learningreinforcement-learning Reinforcement Learning Self-Supervised Learning

Methods 이 논문이 사용한 방법론

Contrastive Learning 설명 없음

ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL

2026-06-02 · Yikang Gui, Bikramjit Banerjee, Prashant Doshi arxiv

Reward transfer in Inverse Reinforcement Learning (IRL) is unreliable when policies must generalize to unseen combinations of environment dynamics and task goals. We propose Factorized Contrastive Abstractions for Transf…

Reinforcement Learning Continuous Control

Learning Markov State Abstractions for Deep Reinforcement Learning

2021-06-08 · NeurIPS 2021 12 · Cameron Allen, Neev Parikh, Omer Gottesman, George Konidaris

A fundamental assumption of reinforcement learning in Markov decision processes (MDPs) is that the relevant decision process is, in fact, Markov. However, when MDPs have rich observations, agents typically learn by way o…

continuous-control Continuous Control Contrastive Learning Deep Reinforcement Learning+3

Multi-Agent Transfer Learning via Temporal Contrastive Learning

2024-06-03 · Weihao Zeng, Joseph Campbell, Simon Stepputtis, Katia Sycara

This paper introduces a novel transfer learning framework for deep multi-agent reinforcement learning. The approach automatically combines goal-conditioned policies with temporal contrastive learning to discover meaningf…

Contrastive Learning Multi-agent Reinforcement Learning Transfer Learning

State Abstractions for Lifelong Reinforcement Learning

2018-07-01 · ICML 2018 7 · David Abel, Dilip Arumugam, Lucas Lehnert, Michael Littman

In lifelong reinforcement learning, agents must effectively transfer knowledge across tasks while simultaneously addressing exploration, credit assignment, and generalization. State abstraction can help overcome the…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Theory of Abstraction in Reinforcement Learning

2022-03-01 · David Abel

Reinforcement learning defines the problem facing agents that learn to make good decisions through action and observation alone. To be effective problem solvers, such agents must efficiently explore vast worlds, assign c…