Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces

2025-07-28 · Saket Tiwari, Omer Gottesman, George Konidaris arxiv

Advances in reinforcement learning (RL) have led to its successful application in complex tasks with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state and action spaces. We propose building a theoretical understanding of continuous state and action spaces by employing a geometric lens to understand the locally attained set of states. The set of all parametrised policies learnt through a semi-gradient based approach induces a set of attainable states in RL. We show that the training dynamics of a two-layer neural policy induce a low dimensional manifold of attainable states embedded in the high-dimensional nominal state space trained using an actor-critic algorithm. We prove that, under certain conditions, the dimensionality of this manifold is of the order of the dimensionality of the action space. This is the first result of its kind, linking the geometry of the state space to the dimensionality of the action space. We empirically corroborate this upper bound for four MuJoCo environments and also demonstrate the results in a toy environment with varying dimensionality. We also show the applicability of this theoretical result by introducing a local manifold learning layer to the policy and value function networks to improve the performance in control environments with very high degrees of freedom by changing one layer of the neural network to learn sparse representations.

📄 PDF Abstract BibTeX arXiv:2507.20853

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

On the Geometry of Reinforcement Learning in Continuous State and Action Spaces

2022-12-29 · Saket Tiwari, Omer Gottesman, George Konidaris

Advances in reinforcement learning have led to its successful application in complex tasks with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state and a…

MuJoCo reinforcement-learning Reinforcement Learning (RL)

Zooming for Efficient Model-Free Reinforcement Learning in Metric Spaces

2020-03-09 · Ahmed Touati, Adrien Ali Taiga, Marc G. Bellemare

Despite the wealth of research into provably efficient reinforcement learning algorithms, most works focus on tabular representation and thus struggle to handle exponentially or infinitely large state-action spaces. In t…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learning

2025-11-11 · Zhihao Lin arxiv

Gaussian policies have dominated continuous control in deep reinforcement learning (RL), yet they suffer from a fundamental mismatch: their unbounded support requires ad-hoc squashing functions that distort the geometry …

Reinforcement Learning Continuous Control

Investigating Reinforcement Learning Agents for Continuous State Space Environments

2017-08-08 · David Von Dollen

Given an environment with continuous state spaces and discrete actions, we investigate using a Double Deep Q-learning Reinforcement Agent to find optimal policies using the LunarLander-v2 OpenAI gym environment.

OpenAI Gym Q-Learning reinforcement-learning Reinforcement Learning+1

Deep Reinforcement Learning in Parameterized Action Space

2015-11-13 · Matthew Hausknecht, Peter Stone

Recent work has shown that deep neural networks are capable of approximating both value functions and policies in reinforcement learning domains featuring continuous state and action spaces. However, to the best of our k…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)