Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method

2020-10-15 · Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, Ding Zhao

This paper studies the constrained/safe reinforcement learning (RL) problem with sparse indicator signals for constraint violations. We propose a model-based approach to enable RL agents to effectively explore the environment with unknown system dynamics and environment constraints given a significantly small number of violation budgets. We employ the neural network ensemble model to estimate the prediction uncertainty and use model predictive control as the basic control framework. We propose the robust cross-entropy method to optimize the control sequence considering the model uncertainty and constraints. We evaluate our methods in the Safety Gym environment. The results show that our approach learns to complete the tasks with a much smaller number of constraint violations than state-of-the-art baselines. Additionally, we are able to achieve several orders of magnitude better sample efficiency when compared with constrained model-free RL approaches. The code is available at \url{https://github.com/liuzuxin/safe-mbrl}.

📄 PDF Abstract BibTeX arXiv:2010.07968

Code (1)

liuzuxin/safe-mbrl 공식 구현 pytorch

Tasks

Model-based Reinforcement LearningModel Predictive Controlreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Constrained Cross-Entropy Method for Safe Reinforcement Learning

2018-12-01 · NeurIPS 2018 12 · Min Wen, Ufuk Topcu

We study a safe reinforcement learning problem in which the constraints are defined as the expected cost over finite-length trajectories. We propose a constrained cross-entropy-based method to solve this problem. The met…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL

2025-10-25 · Guojian Zhan, Likun Wang, Pengcheng Wang, Feihong Zhang 외 arxiv

Maximum entropy has become a mainstream off-policy reinforcement learning (RL) framework for balancing exploitation and exploration. However, two bottlenecks still limit further performance improvement: (1) non-stationar…

Reinforcement Learning OpenAI Gym

Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization

2025-06-12 · Pierre-François Massiani, Alexander von Rohr, Lukas Haverbeck, Sebastian Trimpe

Despite the many recent advances in reinforcement learning (RL), the question of learning policies that robustly satisfy state constraints under unknown disturbances remains open. In this paper, we offer a new perspectiv…

Reinforcement Learning (RL)

Constrained Attractor Selection Using Deep Reinforcement Learning

2019-09-23 · Xue-She Wang, James D. Turner, Brian P. Mann

This paper describes an approach for attractor selection (or multi-stability control) in nonlinear dynamical systems with constrained actuation. Attractor selection is obtained using two different deep reinforcement lear…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learning

2022-03-22 · Fan Ding, Yeiang Xue

Inverse Reinforcement Learning (IRL) is a powerful way of learning from demonstrations. In this paper, we address IRL problems with the availability of prior knowledge that optimal policies will never violate certain con…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)