Uncertainty-aware Model-based Policy Optimization
Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches. However, existing model-based methods are vulnerable to model bias, which leads to poor generalization and asymptotic performance compared to model-free counterparts. In addition, they are typically based on the model predictive control (MPC) framework, which not only is computationally inefficient at decision time but also does not enable policy transfer due to the lack of an explicit policy representation. In this paper, we propose a novel uncertainty-aware model-based policy optimization framework which solves those issues. In this framework, the agent simultaneously learns an uncertainty-aware dynamics model and optimizes the policy according to these learned models. In the optimization step, the policy gradient is computed by automatic differentiation through the models. With respect to sample efficiency alone, our approach shows promising results on challenging continuous control benchmarks with competitive asymptotic performance and significantly lower sample complexity than state-of-the-art baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlmodelModel-based Reinforcement LearningModel Predictive ControlReinforcement LearningSimilar Papers 제목 키워드 기반
Uncertainty-Aware Policy Optimization: A Robust, Adaptive Trust Region Approach
In order for reinforcement learning techniques to be useful in real-world decision making processes, they must be able to produce robust performance from limited data. Deep policy optimization methods have achieved impre…
Decision MakingPPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration
Proximal Policy Optimization (PPO) is a highly popular policy-based deep reinforcement learning (DRL) approach. However, we observe that the homogeneous exploration process in PPO could cause an unexpected stability issu…
continuous-controlContinuous ControlDeep Reinforcement Learningreinforcement-learning+2SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions): some lead to consistent, semantically similar answers, while others yield diverse or contradictory outputs. This variati…
MathMathematical ReasoningSHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor's sen…
Reinforcement LearningDeep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimization
Model-based reinforcement learning algorithms tend to achieve higher sample efficiency than model-free methods. However, due to the inevitable errors of learned models, model-based methods struggle to achieve the same as…
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)