Parameter Transfer Unit for Deep Neural Networks
Parameters in deep neural networks which are trained on large-scale databases can generalize across multiple domains, which is referred as "transferability". Unfortunately, the transferability is usually defined as discrete states and it differs with domains and network architectures. Existing works usually heuristically apply parameter-sharing or fine-tuning, and there is no principled approach to learn a parameter transfer strategy. To address the gap, a parameter transfer unit (PTU) is proposed in this paper. The PTU learns a fine-grained nonlinear combination of activations from both the source and the target domain networks, and subsumes hand-crafted discrete transfer states. In the PTU, the transferability is controlled by two gates which are artificial neurons and can be learned from data. The PTU is a general and flexible module which can be used in both CNNs and RNNs. Experiments are conducted with various network architectures and multiple transfer domain pairs. Results demonstrate the effectiveness of the PTU as it outperforms heuristic parameter-sharing and fine-tuning in most settings.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Parametric Rectified Power Sigmoid Units: Learning Nonlinear Neural Transfer Analytical Forms
The paper proposes representation functionals in a dual paradigm where learning jointly concerns both linear convolutional weights and parametric forms of nonlinear activation functions. The nonlinear forms proposed for …
BIG-bench Machine LearningParameter Transfer Extreme Learning Machine based on Projective Model
Recent years, transfer learning has attracted much attention in the community of machine learning. In this paper, we mainly focus on the tasks of parameter transfer under the framework of extreme learning machine (ELM). …
Domain Adaptationfeature selectionTransfer LearningUnit-level surprise in neural networks
To adapt to changes in real-world data distributions, neural networks must update their parameters. We argue that unit-level surprise should be useful for: (i) determining which few parameters should update to adapt quic…
StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
Real-time strategy games have been an important field of game artificial intelligence in recent years. This paper presents a reinforcement learning and curriculum transfer learning method to control multiple units in Sta…
Real-Time Strategy Gamesreinforcement-learningReinforcement LearningReinforcement Learning (RL)+2HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML
Hyperparameter optimization (HPO) is a core problem for the machine learning community and remains largely unsolved due to the significant computational resources required to evaluate hyperparameter configurations. As a …
Hyperparameter OptimizationTransfer Learning