Offline Reinforcement Learning with Penalized Action Noise Injection
Offline reinforcement learning (RL) optimizes a policy using only a fixed dataset, making it a practical approach in scenarios where interaction with the environment is costly. Due to this limitation, generalization ability is key to improving the performance of offline RL algorithms, as demonstrated by recent successes of offline RL with diffusion models. However, it remains questionable whether such diffusion models are necessary for highly performing offline RL algorithms, given their significant computational requirements during inference. In this paper, we propose Penalized Action Noise Injection (PANI), a method that simply enhances offline learning by utilizing noise-injected actions to cover the entire action space, while penalizing according to the amount of noise injected. This approach is inspired by how diffusion models have worked in offline RL algorithms. We provide a theoretical foundation for this method, showing that offline RL algorithms with such noise-injected actions solve a modified Markov Decision Process (MDP), which we call the noisy action MDP. PANI is compatible with a wide range of existing off-policy and offline RL algorithms, and despite its simplicity, it demonstrates significant performance improvements across various benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningOffline RLSimilar Papers 제목 키워드 기반
Exclusively Penalized Q-learning for Offline Reinforcement Learning
Constraint-based offline reinforcement learning (RL) involves policy constraints or imposing penalties on the value function to mitigate overestimation errors caused by distributional shift. This paper focuses on a limit…
Offline RLQ-Learningreinforcement-learningReinforcement Learning+1GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
Offline Reinforcement Learning (Offline RL) presents challenges of learning effective decision-making policies from static datasets without any online interactions. Data augmentation techniques, such as noise injection a…
Data AugmentationDecision MakingOffline RLreinforcement-learning+1Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
We study the problem of safe offline reinforcement learning (RL), the goal is to learn a policy that maximizes long-term reward while satisfying safety constraints given only offline data, without further interaction wit…
Offline RLQ-Learningreinforcement-learningReinforcement Learning+1Value Penalized Q-Learning for Recommender Systems
Scaling reinforcement learning (RL) to recommender systems (RS) is promising since maximizing the expected cumulative rewards for RL agents meets the objective of RS, i.e., improving customers' long-term satisfaction. A …
Offline RLQ-LearningRecommendation SystemsReinforcement Learning (RL)Differentially Private Deep Model-Based Reinforcement Learning
We address private deep offline reinforcement learning (RL), where the goal is to train a policy on standard control tasks that is differentially private (DP) with respect to individual trajectories in the dataset. To ac…
continuous-controlContinuous ControlmodelModel-based Reinforcement Learning+3