Online Policy Distillation with Decision-Attention
Policy Distillation (PD) has become an effective method to improve deep reinforcement learning tasks. The core idea of PD is to distill policy knowledge from a teacher agent to a student agent. However, the teacher-student framework requires a well-trained teacher model which is computationally expensive.In the light of online knowledge distillation, we study the knowledge transfer between different policies that can learn diverse knowledge from the same environment.In this work, we propose Online Policy Distillation (OPD) with Decision-Attention (DA), an online learning framework in which different policies operate in the same environment to learn different perspectives of the environment and transfer knowledge to each other to obtain better performance together. With the absence of a well-performance teacher policy, the group-derived targets play a key role in transferring group knowledge to each student policy. However, naive aggregation functions tend to cause student policies quickly homogenize. To address the challenge, we introduce the Decision-Attention module to the online policies distillation framework. The Decision-Attention module can generate a distinct set of weights for each policy to measure the importance of group members. We use the Atari platform for experiments with various reinforcement learning algorithms, including PPO and DQN. In different tasks, our method can perform better than an independent training policy on both PPO and DQN algorithms. This suggests that our OPD-DA can transfer knowledge between different policies well and help agents obtain more rewards.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningKnowledge DistillationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
Safe Reinforcement Learning (RL) aims to find a policy that achieves high rewards while satisfying cost constraints. When learning from scratch, safe RL agents tend to be overly conservative, which impedes exploration an…
Autonomous DrivingDecision Makingreinforcement-learningReinforcement Learning+2Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning
Traditional approaches to RL have focused on learning decision policies directly from episodic decisions, while slowly and implicitly learning the semantics of compositional representations needed for generalization. Whi…
Few-Shot LearningLanguage ModellingSelf-LearningTransfer LearningIPD: Boosting Sequential Policy with Imaginary Planning Distillation in Offline Reinforcement Learning
Decision transformer based sequential policies have emerged as a powerful paradigm in offline reinforcement learning (RL), yet their efficacy remains constrained by the quality of static datasets and inherent architectur…
Reinforcement LearningOffline RLMSVIPER: Improved Policy Distillation for Reinforcement-Learning-Based Robot Navigation
We present Multiple Scenario Verifiable Reinforcement Learning via Policy Extraction (MSVIPER), a new method for policy distillation to decision trees for improved robot navigation. MSVIPER learns an "expert" policy usin…
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Neural-to-Tree Policy Distillation with Policy Improvement Criterion
While deep reinforcement learning has achieved promising results in challenging decision-making tasks, the main bones of its success --- deep neural networks are mostly black-boxes. A feasible way to gain insight into a …
Decision MakingDeep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)