Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning. Our approach stems from the idea that the agent's experience in the source domain should look similar to its experience in the target domain. Building off of a probabilistic view of RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function. This modified reward function is simple to estimate by learning auxiliary classifiers that distinguish source-domain transitions from target-domain transitions. Intuitively, the modified reward function penalizes the agent for visiting states and taking actions in the source domain which are not possible in the target domain. Said another way, the agent is penalized for transitions that would indicate that the agent is interacting with the source domain, rather than the target domain. Our approach is applicable to domains with continuous states and actions and does not require learning an explicit model of the dynamics. On discrete and continuous control tasks, we illustrate the mechanics of our approach and demonstrate its scalability to high-dimensional tasks.
Code (2)
Tasks
continuous-controlContinuous ControlDomain Adaptationreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning
Cross-domain offline reinforcement learning leverages source domain data with diverse transition dynamics to alleviate the data requirement for the target domain. However, simply merging the data of two domains leads to …
reinforcement-learningDADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning
Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adapta…
Representation LearningReinforcement LearningDomain AdaptationDecoupling Dynamics and Reward for Transfer Learning
Current reinforcement learning (RL) methods can successfully learn single tasks but often generalize poorly to modest perturbations in task domain or training procedure. In this work, we present a decoupled learning stra…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Transfer LearningModel-Based Reinforcement Learning with Multi-Task Offline Pretraining
Pretraining reinforcement learning (RL) models on offline datasets is a promising way to improve their training efficiency in online tasks, but challenging due to the inherent mismatch in dynamics and behaviors across va…
Knowledge DistillationModel-based Reinforcement Learningreinforcement-learningReinforcement Learning+1Cross-Domain Policy Adaptation by Capturing Representation Mismatch
It is vital to learn effective policies that can be transferred to different domains with dynamics discrepancies in reinforcement learning (RL). In this paper, we consider dynamics adaptation settings where there exists …
Reinforcement Learning (RL)Representation Learning