Operator Deep Q-Learning: Zero-Shot Reward Transferring in Reinforcement Learning
Reinforcement learning (RL) has drawn increasing interests in recent years due to its tremendous success in various applications. However, standard RL algorithms can only be applied for single reward function, and cannot adapt to an unseen reward function quickly. In this paper, we advocate a general operator view of reinforcement learning, which enables us to directly approximate the operator that maps from reward function to value function. The benefit of learning the operator is that we can incorporate any new reward function as input and attain its corresponding value function in a zero-shot manner. To approximate this special type of operator, we design a number of novel operator neural network architectures based on its theoretical properties. Our design of operator networks outperform the existing methods and the standard design of general purpose operator network, and we demonstrate the benefit of our operator deep Q-learning framework in several tasks including reward transferring for offline policy evaluation (OPE) and reward transferring for offline policy optimization in a range of tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIF
In this work, we address the problem of directing the text generation of a language model (LM) towards a desired behavior, aligning the generated text with the preferences of the human operator. We propose using another,…
AttributeDiversityLanguage ModelingLanguage Modelling+1A Unified Framework for Zero-Shot Reinforcement Learning
Zero-shot reinforcement learning (RL) has emerged as a setting for developing general agents, capable of solving downstream tasks without additional training or planning at test-time. While conventional RL optimizes poli…
Reinforcement LearningPEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation
In preference-based Reinforcement Learning (RL), obtaining a large number of preference labels are both time-consuming and costly. Furthermore, the queried human preferences cannot be utilized for the new tasks. In this …
Offline RLReinforcement Learning (RL)Tackling the Zero-Shot Reinforcement Learning Loss Directly
Zero-shot reinforcement learning (RL) methods aim at instantly producing a behavior for an RL task in a given environment, from a description of the reward function. These methods are usually tested by evaluating their a…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings
Can we pre-train a generalist agent from a large amount of unlabeled offline trajectories such that it can be immediately adapted to any new downstream tasks in a zero-shot manner? In this work, we present a functional r…
DiversityOffline RLreinforcement-learningReinforcement Learning