DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing
Model-based reinforcement learning (MBRL) has gained much attention for its ability to learn complex behaviors in a sample-efficient way: planning actions by generating imaginary trajectories with predicted rewards. Despite its success, we found that surprisingly, reward prediction is often a bottleneck of MBRL, especially for sparse rewards that are challenging (or even ambiguous) to predict. Motivated by the intuition that humans can learn from rough reward estimates, we propose a simple yet effective reward smoothing approach, DreamSmooth, which learns to predict a temporally-smoothed reward, instead of the exact reward at the given timestep. We empirically show that DreamSmooth achieves state-of-the-art performance on long-horizon sparse-reward tasks both in sample efficiency and final performance without losing performance on common benchmarks, such as Deepmind Control Suite and Atari benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing
As reinforcement learning (RL) has achieved great success and been even adopted in safety-critical domains such as autonomous vehicles, a range of empirical studies have been conducted to improve its robustness against a…
Atari GamesAutonomous Vehiclesreinforcement-learningReinforcement Learning+1Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation
Preference-based reinforcement learning (PbRL) has shown impressive capabilities in training agents without reward engineering. However, a notable limitation of PbRL is its dependency on substantial human feedback. This …
reinforcement-learningReinforcement LearningIterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique that aligns language models closely with human-centric values. The initial phase of RLHF involves learning human values using a reward model from r…
K-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learning
We propose a simple yet effective alternative to reward normalization in policy gradient reinforcement learning by integrating a 1D Kalman filter for online reward estimation. Instead of relying on fixed heuristics, our …
Reinforcement LearningLearning Guidance Rewards with Trajectory-space Smoothing
Long-term temporal credit assignment is an important challenge in deep reinforcement learning (RL). It refers to the ability of the agent to attribute actions to consequences that may occur after a long time interval. Ex…
AttributeDeep Reinforcement LearningQ-LearningReinforcement Learning (RL)