Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
Offline reinforcement learning (offline RL) offers a promising framework for developing control strategies in chemical process systems using historical data, without the risks or costs of online experimentation. This work investigates the application of offline RL to the safe and efficient control of an exothermic polymerisation continuous stirred-tank reactor. We introduce a Gymnasium-compatible simulation environment that captures the reactor's nonlinear dynamics, including reaction kinetics, energy balances, and operational constraints. The environment supports three industrially relevant scenarios: startup, grade change down, and grade change up. It also includes reproducible offline datasets generated from proportional-integral controllers with randomised tunings, providing a benchmark for evaluating offline RL algorithms in realistic process control tasks. We assess behaviour cloning and implicit Q-learning as baseline algorithms, highlighting the challenges offline agents face, including steady-state offsets and degraded performance near setpoints. To address these issues, we propose a novel deployment-time safety layer that performs gradient-based action correction using input convex neural networks (PICNNs) as learned cost models. The PICNN enables real-time, differentiable correction of policy actions by descending a convex, state-conditioned cost surface, without requiring retraining or environment interaction. Experimental results show that offline RL, particularly when combined with convex action correction, can outperform traditional control approaches and maintain stability across all scenarios. These findings demonstrate the feasibility of integrating offline RL with interpretable and safety-aware corrections for high-stakes chemical process control, and lay the groundwork for more reliable data-driven automation in industrial systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningOffline RLSimilar Papers 제목 키워드 기반
Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inferen…
Reinforcement LearningOffline RLFOSP: Fine-tuning Offline Safe Policy through World Models
Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize …
Model-based Reinforcement LearningOffline RLReinforcement Learning (RL)Safe Reinforcement LearningConditional Sequence Modeling for Safe Reinforcement Learning
Offline safe reinforcement learning (RL) aims to learn policies from a fixed dataset while maximizing performance under cumulative cost constraints. In practice, deployment requirements often vary across scenarios, neces…
Reinforcement LearningReinforcement Learning with Ensemble Model Predictive Safety Certification
Reinforcement learning algorithms need exploration to learn. However, unsupervised exploration prevents the deployment of such algorithms on safety-critical tasks and limits real-world deployment. In this paper, we propo…
Deep Reinforcement LearningmodelModel Predictive Controlreinforcement-learning+1Constrained Decision Transformer for Offline Safe Reinforcement Learning
Safe reinforcement learning (RL) trains a constraint satisfaction policy by interacting with the environment. We aim to tackle a more challenging problem: learning a safe policy from an offline dataset. We study the offl…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning