Net2Net: Accelerating Learning via Knowledge Transfer
We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net. During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network. Our techniques are based on the concept of function-preserving transformations between neural network specifications. This differs from previous approaches to pre-training that altered the function represented by a neural net when adding layers to it. Using our knowledge transfer mechanism to add depth to Inception modules, we demonstrate a new state of the art accuracy rating on the ImageNet dataset.
Code (3)
Tasks
Transfer LearningSimilar Papers 제목 키워드 기반
Accelerating Deep Unsupervised Domain Adaptation with Transfer Channel Pruning
Deep unsupervised domain adaptation (UDA) has recently received increasing attention from researchers. However, existing methods are computationally intensive due to the computation cost of Convolutional Neural Networks …
Domain AdaptationTransfer LearningUnsupervised Domain AdaptationAccelerating Dependency Graph Learning from Heterogeneous Categorical Event Streams via Knowledge Transfer
Dependency graph, as a heterogeneous graph representing the intrinsic relationships between different pairs of system entities, is essential to many data analysis applications, such as root cause diagnosis, intrusion det…
Graph LearningIntrusion DetectionTransfer LearningREPAINT: Knowledge Transfer in Deep Reinforcement Learning
Accelerating learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target t…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach
Peer-to-peer knowledge transfer in distributed environments has emerged as a promising method since it could accelerate learning and improve team-wide performance without relying on pre-trained teachers in deep reinforce…
Deep Reinforcement LearningReinforcement LearningTransfer LearningFast and Efficient DNN Deployment via Deep Gaussian Transfer Learning
Deep neural networks (DNNs) have been widely used recently while their hardware deployment optimizations are very time-consuming and the historical deployment knowledge is not utilized efficiently. In this paper, to …
Gaussian ProcessesTransfer Learning