A Robust Policy Bootstrapping Algorithm for Multi-objective Reinforcement Learning in Non-stationary Environments
Multi-objective Markov decision processes are a special kind of multi-objective optimization problem that involves sequential decision making while satisfying the Markov property of stochastic processes. Multi-objective reinforcement learning methods address this problem by fusing the reinforcement learning paradigm with multi-objective optimization techniques. One major drawback of these methods is the lack of adaptability to non-stationary dynamics in the environment. This is because they adopt optimization procedures that assume stationarity to evolve a coverage set of policies that can solve the problem. This paper introduces a developmental optimization approach that can evolve the policy coverage set while exploring the preference space over the defined objectives in an online manner. We propose a novel multi-objective reinforcement learning algorithm that can robustly evolve a convex coverage set of policies in an online manner in non-stationary environments. We compare the proposed algorithm with two state-of-the-art multi-objective reinforcement learning algorithms in stationary and non-stationary environments. Results showed that the proposed algorithm significantly outperforms the existing algorithms in non-stationary environments while achieving comparable results in stationary environments.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingMulti-Objective Reinforcement Learningreinforcement-learningReinforcement LearningSequential Decision MakingSimilar Papers 제목 키워드 기반
Meta-Gradient Reinforcement Learning with an Objective Discovered Online
Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with r…
Deep Reinforcement LearningQ-Learningreinforcement-learningReinforcement Learning+1Safe Policy Improvement with Soft Baseline Bootstrapping
Batch Reinforcement Learning (Batch RL) consists in training a policy using trajectories collected with another policy, called the behavioural policy. Safe policy improvement (SPI) provides guarantees with high probabili…
Reinforcement LearningMulti-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning (RL) setting. We consider the scenario where: (i) we have a dataset collected under a known baseline policy, (…
Reinforcement Learning (RL)Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q…
continuous-controlContinuous ControlQ-LearningReinforcement LearningBootstrapping Fitted Q-Evaluation for Off-Policy Inference
Bootstrapping provides a flexible and effective approach for assessing the quality of batch reinforcement learning, yet its theoretical property is less understood. In this paper, we study the use of bootstrapping in off…
Off-policy evaluation