Preserving Expert-Level Privacy in Offline Reinforcement Learning
The offline reinforcement learning (RL) problem aims to learn an optimal policy from historical data collected by one or more behavioural policies (experts) by interacting with an environment. However, the individual experts may be privacy-sensitive in that the learnt policy may retain information about their precise choices. In some domains like personalized retrieval, advertising and healthcare, the expert choices are considered sensitive data. To provably protect the privacy of such experts, we propose a novel consensus-based expert-level differentially private offline RL training approach compatible with any existing offline RL algorithm. We prove rigorous differential privacy guarantees, while maintaining strong empirical performance. Unlike existing work in differentially private RL, we supplement the theory with proof-of-concept experiments on classic RL environments featuring large continuous state spaces, demonstrating substantial improvements over a natural baseline across multiple tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)RetrievalSimilar Papers 제목 키워드 기반
Order-Optimal Instance-Dependent Bounds for Offline Reinforcement Learning with Preference Feedback
We consider offline reinforcement learning (RL) with preference feedback in which the implicit reward is a linear function of an unknown parameter. Given an offline dataset, our objective consists in ascertaining the opt…
Offline RLReinforcement Learning (RL)BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learning
Offline inverse reinforcement learning (IRL) aims to recover a reward function that explains expert behavior using only fixed demonstration data, without any additional online interaction. We propose BiCQL-ML, a policy-f…
Reinforcement LearningOffline RLWhen Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurat…
Autonomous Drivingcontinuous-controlContinuous ControlD4RL+3ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Games
Offline learning has become widely used due to its ability to derive effective policies from offline datasets gathered by expert demonstrators without interacting with the environment directly. Recent research has explor…
Imitation LearningDifferentially Private Deep Model-Based Reinforcement Learning
We address private deep offline reinforcement learning (RL), where the goal is to train a policy on standard control tasks that is differentially private (DP) with respect to individual trajectories in the dataset. To ac…
continuous-controlContinuous ControlmodelModel-based Reinforcement Learning+3