RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch
Training deep reinforcement learning (DRL) models usually requires high computation costs. Therefore, compressing DRL models possesses immense potential for training acceleration and model deployment. However, existing methods that generate small models mainly adopt the knowledge distillation-based approach by iteratively training a dense network. As a result, the training process still demands massive computing resources. Indeed, sparse training from scratch in DRL has not been well explored and is particularly challenging due to non-stationarity in bootstrap training. In this work, we propose a novel sparse DRL training framework, "the Rigged Reinforcement Learning Lottery" (RLx2), which builds upon gradient-based topology evolution and is capable of training a sparse DRL model based entirely on a sparse network. Specifically, RLx2 introduces a novel multi-step TD target mechanism with a dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration in sparse models. It also reaches state-of-the-art sparse training performance in several tasks, showing 7.5\times-20\times model compression with less than 3% performance degradation and up to 20\times and 50\times FLOPs reduction for training and inference, respectively.
Code (1)
Tasks
Continuous ControlDeep Reinforcement LearningKnowledge DistillationModel Compressionreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dynamic Sparse Training for Deep Reinforcement Learning
Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training time for dense neural networks to achieve good performance. Hence, prohibitive…
continuous-controlContinuous ControlDecision MakingDeep Reinforcement Learning+4Learning by Playing - Solving Sparse Reward Tasks from Scratch
We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in the context of Reinforcement Learning (RL). SAC-X enables learning of complex behaviors - from scratch - in the presence of multiple sparse rewar…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)SchedulingMSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization
Pre-trained models have been a foundational approach in speech recognition, albeit with associated additional costs. In this study, we propose a regularization technique that facilitates the training of visual and audio-…
Audio-Visual Speech Recognitionspeech-recognitionSpeech RecognitionVisual Speech RecognitionPhasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning
It has been a recent trend to leverage the power of supervised learning (SL) towards more effective reinforcement learning (RL) methods. We propose a novel phasic approach by alternating online RL and offline SL for tack…
reinforcement-learningReinforcement Learning (RL)Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance
Generative adversarial networks (GANs) have received an upsurging interest since being proposed due to the high quality of the generated data. While achieving increasingly impressive results, the resource demands associa…
Model Compression