Confounding Robust Continuous Control via Automatic Reward Shaping
Reward shaping has been applied widely to accelerate Reinforcement Learning (RL) agents' training. However, a principled way of designing effective reward shaping functions, especially for complex continuous control problems, remains largely under-explained. In this work, we propose to automatically learn a reward shaping function for continuous control problems from offline datasets, potentially contaminated by unobserved confounding variables. Specifically, our method builds upon the recently proposed causal Bellman equation to learn a tight upper bound on the optimal state values, which is then used as the potentials in the Potential-Based Reward Shaping (PBRS) framework. Our proposed reward shaping algorithm is tested with Soft-Actor-Critic (SAC) on multiple commonly used continuous control benchmarks and exhibits strong performance guarantees under unobserved confounders. More broadly, our work marks a solid first step towards confounding robust continuous control from a causal perspective. Code for training our reward shaping functions can be found at https://github.com/mateojuliani/confounding_robust_cont_control.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningContinuous ControlSimilar Papers 제목 키워드 기반
ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization
Reward shaping is critical in reinforcement learning (RL), particularly for complex tasks where sparse rewards can hinder learning. However, choosing effective shaping rewards from a set of reward functions in a computat…
continuous-controlContinuous ControlModel SelectionReinforcement Learning (RL)Automatic Reward Shaping from Confounded Offline Data
A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make o…
Atari GamesDeep Reinforcement LearningQ-LearningDense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations
Many continuous control problems can be formulated as sparse-reward reinforcement learning (RL) tasks. In principle, online RL methods can automatically explore the state space to solve each new task. However, discoverin…
continuous-controlContinuous ControlReinforcement Learning (RL)Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learning
We present AIRS: Automatic Intrinsic Reward Shaping that intelligently and adaptively provides high-quality intrinsic rewards to enhance exploration in reinforcement learning (RL). More specifically, AIRS selects shaping…
BenchmarkingDeep Reinforcement Learningreinforcement-learningReinforcement Learning+1Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learning
Reinforcement learning, which acquires a policy maximizing long-term rewards, has been actively studied. Unfortunately, this learning type is too slow and difficult to use in practical situations because the state-action…
AI Agentreinforcement-learningReinforcement LearningReinforcement Learning (RL)