MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator
Offline reinforcement learning (RL) faces a significant challenge of distribution shift. Model-free offline RL penalizes the Q value for out-of-distribution (OOD) data or constrains the policy closed to the behavior policy to tackle this problem, but this inhibits the exploration of the OOD region. Model-based offline RL, which uses the trained environment model to generate more OOD data and performs conservative policy optimization within that model, has become an effective method for this problem. However, the current model-based algorithms rarely consider agent robustness when incorporating conservatism into policy. Therefore, the new model-based offline algorithm with a conservative Bellman operator (MICRO) is proposed. This method trades off performance and robustness via introducing the robust Bellman operator into the algorithm. Compared with previous model-based algorithms with robust adversarial models, MICRO can significantly reduce the computation cost by only choosing the minimal Q value in the state uncertainty set. Extensive experiments demonstrate that MICRO outperforms prior RL algorithms in offline RL benchmark and is considerably robust to adversarial perturbations.
Code (1)
Tasks
Offline RLreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Peng's Q($λ$) for Conservative Value Estimation in Offline Reinforcement Learning
We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q($λ$) (CPQL). Our algorithm adapts the Peng's Q($λ$) (PQL) operator for conservative value estimation as an alternati…
Reinforcement LearningOffline RLImplicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning
Offline reinforcement learning promises policy improvement from logged interaction data alone, yet state-of-the-art algorithms remain vulnerable to value over-estimation and to violations of domain knowledge such as mono…
Q-LearningConservative Offline Distributional Reinforcement Learning
Many reinforcement learning (RL) problems in practice are offline, learning purely from observational data. A key challenge is how to ensure the learned policy is safe, which requires quantifying the risk associated with…
D4RLDistributional Reinforcement LearningMuJoCoOffline RL+4Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without further environment interaction. A key challenge is the distribution shift between the learned and behavior policies, leadi…
Reinforcement LearningOffline RLSelecting Offline Reinforcement Learning Algorithms for Stochastic Network Control
Offline Reinforcement Learning (RL) is a promising approach for next-generation wireless networks, where online exploration is unsafe and large amounts of operational data can be reused across the model lifecycle. Howeve…
Reinforcement LearningOffline RL