Self-Supervised Online Reward Shaping in Sparse-Reward Environments
We introduce Self-supervised Online Reward Shaping (SORS), which aims to improve the sample efficiency of any RL algorithm in sparse-reward environments by automatically densifying rewards. The proposed framework alternates between classification-based reward inference and policy update steps -- the original sparse reward provides a self-supervisory signal for reward inference by ranking trajectories that the agent observes, while the policy update is performed with the newly inferred, typically dense reward function. We introduce theory that shows that, under certain conditions, this alteration of the reward function will not change the optimal policy of the original MDP, while potentially increasing learning speed significantly. Experimental results on several sparse-reward environments demonstrate that, across multiple domains, the proposed algorithm is not only significantly more sample efficient than a standard RL baseline using sparse rewards, but, at times, also achieves similar sample efficiency compared to when hand-designed dense reward functions are used.
Code (1)
Similar Papers 제목 키워드 기반
ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning
Sparse rewards are a major bottleneck in multi-agent reinforcement learning (MARL), where simultaneous learning induces non-stationarity and makes reward design especially delicate. Reward shaping can accelerate learning…
Multi-agent Reinforcement LearningShaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approach
In many real-world scenarios, reward signal for agents are exceedingly sparse, making it challenging to learn an effective reward function for reward shaping. To address this issue, our approach performs reward shaping n…
Data Augmentationreinforcement-learningReinforcement LearningReward prediction for representation learning and reward shaping
One of the fundamental challenges in reinforcement learning (RL) is the one of data efficiency: modern algorithms require a very large number of training samples, especially compared to humans, for solving environments w…
PredictionReinforcement Learning (RL)Representation LearningORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization
Reward shaping is critical in reinforcement learning (RL), particularly for complex tasks where sparse rewards can hinder learning. However, choosing effective shaping rewards from a set of reward functions in a computat…
continuous-controlContinuous ControlModel SelectionReinforcement Learning (RL)Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning
Reward shaping is a technique in reinforcement learning that addresses the sparse-reward problem by providing more frequent and informative rewards. We introduce a self-adaptive and highly efficient reward shaping mechan…
Continuous ControlDensity Estimationreinforcement-learningReinforcement Learning