Learn Faster and Forget Slower via Fast and Stable Task Adaptation
Training Deep Neural Networks (DNNs) is still highly time-consuming and compute-intensive. It has been shown that adapting a pretrained model may significantly accelerate this process. With a focus on classification, we show that current fine-tuning techniques make the pretrained models catastrophically forget the transferred knowledge even before anything about the new task is learned. Such rapid knowledge loss undermines the merits of transfer learning and may result in a much slower convergence rate compared to when the maximum amount of knowledge is exploited. We investigate the source of this problem from different perspectives and to alleviate it, introduce Fast And Stable Task-adaptation (FAST), an easy to apply fine-tuning algorithm. The paper provides a novel geometric perspective on how the loss landscape of source and target tasks are linked in different transfer learning strategies. We empirically show that compared to prevailing fine-tuning practices, FAST learns the target task faster and forgets the source task slower.
Code (0)
등록된 구현이 없습니다.
Tasks
Transfer LearningSimilar Papers 제목 키워드 기반
Dropout as an Implicit Gating Mechanism For Continual Learning
In recent years, neural networks have demonstrated an outstanding ability to achieve complex learning tasks across various domains. However, they suffer from the "catastrophic forgetting" problem when they face a sequenc…
Continual LearningParameter Identification of DC Motor based on Compound Least Square Method
For forgetting factor least square identification results are prone to volatility of faults, by introducing selection control, this paper proposes a composite method of least square motor parameter identification, when t…
When slower is faster
The slower is faster (SIF) effect occurs when a system performs worse as its components try to do better. Thus, a moderate individual efficiency actually leads to a better systemic performance. The SIF effect takes place…
Ordered Network Analysis of Epistemic Emotions during Collaborative Problem Solving
Investigating how affective states such as confusion and frustration persist and transition during co-situated collaborative problem solving (CPS) is important for understanding the dynamics of epistemic emotions. Howeve…
Tabular and Deep Learning for the Whittle Index
The Whittle index policy is a heuristic that has shown remarkably good performance (with guaranteed asymptotic optimality) when applied to the class of problems known as Restless Multi-Armed Bandit Problems (RMABPs). In …
Deep LearningQ-Learning