Off-Policy Reinforcement Learning with High Dimensional Reward
Conventional off-policy reinforcement learning (RL) focuses on maximizing the expected return of scalar rewards. Distributional RL (DRL), in contrast, studies the distribution of returns with the distributional Bellman operator in a Euclidean space, leading to highly flexible choices for utility. This paper establishes robust theoretical foundations for DRL. We prove the contraction property of the Bellman operator even when the reward space is an infinite-dimensional separable Banach space. Furthermore, we demonstrate that the behavior of high- or infinite-dimensional returns can be effectively approximated using a lower-dimensional Euclidean space. Leveraging these theoretical insights, we propose a novel DRL algorithm that tackles problems which have been previously intractable using conventional reinforcement learning approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Reinforcement Learning without Ground-Truth State
To perform robot manipulation tasks, a low-dimensional state of the environment typically needs to be estimated. However, designing a state estimator can sometimes be difficult, especially in environments with deformable…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Robot ManipulationLearning Intrinsic Symbolic Rewards in Reinforcement Learning
Learning effective policies for sparse objectives is a key challenge in Deep Reinforcement Learning (RL). A common approach is to design task-related dense rewards to improve task learnability. While such rewards are eas…
Deep Reinforcement LearningMuJoCoreinforcement-learningReinforcement Learning+1On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theory
In the context of inverse reinforcement learning (IRL) with a single expert, adversarial inverse reinforcement learning (AIRL) serves as a foundational approach to providing comprehensive and transferable task descriptio…
reinforcement-learningReinforcement LearningSample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks
A recently popular approach to solving reinforcement learning is with data from human preferences. In fact, human preference data are now used with classic reinforcement learning algorithms such as actor-critic methods, …
Off-policy evaluationreinforcement-learningReward prediction for representation learning and reward shaping
One of the fundamental challenges in reinforcement learning (RL) is the one of data efficiency: modern algorithms require a very large number of training samples, especially compared to humans, for solving environments w…
PredictionReinforcement Learning (RL)Representation Learning