Online Abstraction with MDP Homomorphisms for Deep Learning
Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy. In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces. It is based on MDP homomorphisms, a structure-preserving mapping between MDPs. We demonstrate our algorithm's ability to learn abstractions from collected experience and show how to reuse the abstractions to guide exploration in new tasks the agent encounters. Our novel task transfer method outperforms baselines based on a deep Q-network in the majority of our experiments. The source code is at https://github.com/ondrejba/aamas_19.
Code (1)
Tasks
Deep LearningSimilar Papers 제목 키워드 기반
Continuous MDP Homomorphisms and Homomorphic Policy Gradient
Abstraction has been widely studied as a way to improve the efficiency and generalization of reinforcement learning algorithms. In this paper, we study abstraction in the continuous-control setting. We extend the definit…
continuous-controlContinuous ControlPolicy Gradient MethodsReinforcement Learning (RL)+1Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction
How can a scientist use a Reinforcement Learning (RL) algorithm to design experiments over a dynamical system's state space? In the case of finite and Markovian systems, an area called Active Exploration (AE) relaxes the…
Computational EfficiencyReinforcement Learning (RL)scientific discoveryPolicy Gradient Methods in the Presence of Symmetries and State Abstractions
Reinforcement learning (RL) on high-dimensional and complex problems relies on abstraction for improved efficiency and generalization. In this paper, we study abstraction in the continuous-control setting, and extend the…
continuous-controlContinuous ControlPolicy Gradient MethodsReinforcement Learning (RL)+1Delayed homomorphic reinforcement learning for environments with delayed feedback
Reinforcement learning in real-world systems often involves delayed feedback, which breaks the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches cause state-space explosion,…
Reinforcement LearningNeural ring homomorphisms and maps between neural codes
Neural codes are binary codes that are used for information processing and representation in the brain. In previous work, we have shown how an algebraic structure, called the {\it neural ring}, can be used to efficiently…