SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning
The performance of offline reinforcement learning (RL) suffers from the limited size and quality of static datasets. Model-based offline RL addresses this issue by generating synthetic samples through a dynamics model to enhance overall performance. To evaluate the reliability of the generated samples, uncertainty estimation methods are often employed. However, model ensemble, the most commonly used uncertainty estimation method, is not always the best choice. In this paper, we propose a \textbf{S}earch-based \textbf{U}ncertainty estimation method for \textbf{M}odel-based \textbf{O}ffline RL (SUMO) as an alternative. SUMO characterizes the uncertainty of synthetic samples by measuring their cross entropy against the in-distribution dataset samples, and uses an efficient search-based method for implementation. In this way, SUMO can achieve trustworthy uncertainty estimation. We integrate SUMO into several model-based offline RL algorithms including MOPO and Adapted MOReL (AMOReL), and provide theoretical analysis for them. Extensive experimental results on D4RL datasets demonstrate that SUMO can provide more accurate uncertainty estimation and boost the performance of base algorithms. These indicate that SUMO could be a better uncertainty estimator for model-based offline RL when used in either reward penalty or trajectory truncation. Our code is available and will be open-source for further research and development.
Code (0)
등록된 구현이 없습니다.
Tasks
D4RLOffline RLReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Uncertainty Estimation Using Riemannian Model~Dynamics for Offline Reinforcement Learning
Model-based offline reinforcement learning approaches generally rely on bounds of model error. Estimating these bounds is usually achieved through uncertainty estimation methods. In this work, we combine parametric and n…
Autonomous Drivingcontinuous-controlContinuous ControlDecoder+4Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning
For offline reinforcement learning (RL), model-based methods are expected to be data-efficient as they incorporate dynamics models to generate more data. However, due to inevitable model errors, straightforwardly learnin…
D4RLmodelOffline RLreinforcement-learning+3ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems
Offline reinforcement learning (RL) is an effective tool for real-world recommender systems with its capacity to model the dynamic interest of users and its interactive nature. Most existing offline RL recommender system…
Offline RLRecommendation Systemsreinforcement-learningReinforcement Learning+1Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning
Most theoretically motivated work in the offline reinforcement learning setting requires precise uncertainty estimates. This requirement restricts the algorithms derived in that work to the tabular and linear settings wh…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Offline Model-Based Reinforcement Learning with Anti-Exploration
Model-based reinforcement learning (MBRL) algorithms learn a dynamics model from collected data and apply it to generate synthetic trajectories to enable faster learning. This is an especially promising paradigm in offli…
D4RLmodelModel-based Reinforcement LearningOffline RL+3