Reducing Conservativeness Oriented Offline Reinforcement Learning
In offline reinforcement learning, a policy learns to maximize cumulative rewards with a fixed collection of data. Towards conservative strategy, current methods choose to regularize the behavior policy or learn a lower bound of the value function. However, exorbitant conservation tends to impair the policy's generalization ability and degrade its performance, especially for the mixed datasets. In this paper, we propose the method of reducing conservativeness oriented reinforcement learning. On the one hand, the policy is trained to pay more attention to the minority samples in the static dataset to address the data imbalance problem. On the other hand, we give a tighter lower bound of value function than previous methods to discover potential optimal actions. Consequently, our proposed method is able to tackle the skewed distribution of the provided dataset and derive a value function closer to the expected value function. Experimental results demonstrate that our proposed method outperforms the state-of-the-art methods in D4RL offline reinforcement learning evaluation tasks and our own designed mixed datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
D4RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Prescribing Decision Conservativeness in Two-Stage Power Markets: A Distributionally Robust End-to-End Approach
This paper presents an end-to-end framework for calibrating wind power forecast models to minimize operational costs in two-stage power markets, where the first stage involves a distributionally robust optimal power flow…
Uncertainty QuantificationIterative Decomposition of Joint Chance Constraints in OPF
In chance-constrained OPF models, joint chance constraints (JCCs) offer a stronger guarantee on security compared to single chance constraints (SCCs). Using Boole's inequality or its improved versions to decompose JCCs i…
Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
In offline reinforcement learning, the challenge of out-of-distribution (OOD) is pronounced. To address this, existing methods often constrain the learned policy through policy regularization. However, these methods ofte…
D4RLreinforcement-learningReinforcement LearningA Kernel Mean Embedding Approach to Reducing Conservativeness in Stochastic Programming and Control
We apply kernel mean embedding methods to sample-based stochastic optimization and control. Specifically, we use the reduced-set expansion method as a way to discard sampled scenarios. The effect of such constraint remov…
Stochastic OptimizationImproved Off-policy Reinforcement Learning in Biological Sequence Design
Designing biological sequences with desired properties is challenging due to vast search spaces and limited evaluation budgets. Although reinforcement learning methods use proxy models for rapid reward evaluation, insuff…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)