On the Geometry of Reinforcement Learning in Continuous State and Action Spaces
Advances in reinforcement learning have led to its successful application in complex tasks with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state and action spaces. We propose building a theoretical understanding of continuous state and action spaces by employing a geometric lens. Central to our work is the idea that the transition dynamics induce a low dimensional manifold of reachable states embedded in the high-dimensional nominal state space. We prove that, under certain conditions, the dimensionality of this manifold is at most the dimensionality of the action space plus one. This is the first result of its kind, linking the geometry of the state space to the dimensionality of the action space. We empirically corroborate this upper bound for four MuJoCo environments. We further demonstrate the applicability of our result by learning a policy in this low dimensional representation. To do so we introduce an algorithm that learns a mapping to a low dimensional representation, as a narrow hidden layer of a deep neural network, in tandem with the policy using DDPG. Our experiments show that a policy learnt this way perform on par or better for four MuJoCo control suite tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
MuJoCoreinforcement-learningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces
Advances in reinforcement learning (RL) have led to its successful application in complex tasks with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state …
Reinforcement LearningZooming for Efficient Model-Free Reinforcement Learning in Metric Spaces
Despite the wealth of research into provably efficient reinforcement learning algorithms, most works focus on tabular representation and thus struggle to handle exponentially or infinitely large state-action spaces. In t…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learning
Gaussian policies have dominated continuous control in deep reinforcement learning (RL), yet they suffer from a fundamental mismatch: their unbounded support requires ad-hoc squashing functions that distort the geometry …
Reinforcement LearningContinuous ControlInvestigating Reinforcement Learning Agents for Continuous State Space Environments
Given an environment with continuous state spaces and discrete actions, we investigate using a Double Deep Q-learning Reinforcement Agent to find optimal policies using the LunarLander-v2 OpenAI gym environment.
OpenAI GymQ-Learningreinforcement-learningReinforcement Learning+1Deep Reinforcement Learning in Parameterized Action Space
Recent work has shown that deep neural networks are capable of approximating both value functions and policies in reinforcement learning domains featuring continuous state and action spaces. However, to the best of our k…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)