Safe Offline Reinforcement Learning with Real-Time Budget Constraints

2023-06-01 · Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, Dong Wang

Aiming at promoting the safe real-world deployment of Reinforcement Learning (RL), research on safe RL has made significant progress in recent years. However, most existing works in the literature still focus on the online setting where risky violations of the safety budget are likely to be incurred during training. Besides, in many real-world applications, the learned policy is required to respond to dynamically determined safety budgets (i.e., constraint threshold) in real time. In this paper, we target at the above real-time budget constraint problem under the offline setting, and propose Trajectory-based REal-time Budget Inference (TREBI) as a novel solution that models this problem from the perspective of trajectory distribution and solves it through diffusion model planning. Theoretically, we prove an error bound of the estimation on the episodic reward and cost under the offline setting and thus provide a performance guarantee for TREBI. Empirical results on a wide range of simulation tasks and a real-world large-scale advertising application demonstrate the capability of TREBI in solving real-time budget constraint problems under offline settings.

📄 PDF Abstract BibTeX arXiv:2306.00603

Code (1)

qianlin04/safe-offline-rl-with-diffusion-model 공식 구현 jax

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Focus 설명 없음

Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

2026-03-08 · Janaka Chathuranga Brahmanage, Akshat Kumar arxiv

Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balan…

Reinforcement Learning Decision Making

Online Optimization for Offline Safe Reinforcement Learning

2025-10-24 · Yassine Chemingui, Aryan Deshwal, Alan Fern, Thanh Nguyen-Tang 외 arxiv

We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames t…

Reinforcement Learning Offline RL

Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learning

2026-05-04 · Rufeng Chen, Zhaofan Zhang, Zhejiang Yang, Hechang Chen 외 arxiv

Offline safe reinforcement learning often requires policies to adapt at deployment time to safety budgets that vary across episodes or change within a single episode. While diffusion-based planners enable flexible trajec…

Reinforcement Learning

BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion

2022-07-16 · Fanglin Chen, Xiao Liu, Bo Tang, Feiyu Xiong 외

We utilize an offline reinforcement learning (RL) model for sequential targeted promotion in the presence of budget constraints in a real-world business environment. In our application, the mobile app aims to boost custo…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

2026-03-16 · Mumuksh Tayal, Manan Tayal, Ravi Prakash arxiv

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inferen…