Compacting, Picking and Growing for Unforgetting Continual Learning
Continual lifelong learning is essential to many applications. In this paper, we propose a simple but effective approach to continual deep learning. Our approach leverages the principles of deep model compression, critical weights selection, and progressive networks expansion. By enforcing their integration in an iterative manner, we introduce an incremental learning method that is scalable to the number of sequential tasks in a continual learning process. Our approach is easy to implement and owns several favorable characteristics. First, it can avoid forgetting (i.e., learn new tasks while remembering all previous tasks). Second, it allows model expansion but can maintain the model compactness when handling sequential tasks. Besides, through our compaction and selection/expansion mechanism, we show that the knowledge accumulated through learning previous tasks is helpful to build a better model for the new tasks compared to training the models independently with tasks. Experimental results show that our approach can incrementally learn a deep model tackling multiple tasks without forgetting, while the model compactness is maintained with the performance more satisfiable than individual task training.
Code (2)
Tasks
Age And Gender ClassificationContinual LearningFace VerificationFacial Expression Recognition (FER)Incremental LearningLifelong learningModel CompressionSimilar Papers 제목 키워드 기반
PRANC: Pseudo RAndom Networks for Compacting deep models
We demonstrate that a deep model can be reparametrized as a linear combination of several randomly initialized and frozen deep models in the weight space. During training, we seek local minima that reside within the subs…
image-classificationImage ClassificationEXTENDING CONDITIONAL CONVOLUTION STRUCTURES FOR ENHANCING MULTITASKING CONTINUAL LEARNING
Conditional operations have received much attention in recent deep learning studies to facilitate the prediction accuracy of a model. A recent advance toward this direction is the conditional parametric convolutions (Con…
Computational EfficiencyContinual LearningMulti-Task LearningCompacting Deep Neural Networks for Internet of Things: Methods and Applications
Deep Neural Networks (DNNs) have shown great success in completing complex tasks. However, DNNs inevitably bring high computational cost and storage consumption due to the complexity of hierarchical structures, thereby h…
DiversityKnowledge DistillationModel CompressionGradually Compacting Large Language Models for Reasoning Like a Boiling Frog
Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, but their substantial size often demands significant computational resources. To reduce resource consumption and accelerate inference, it …
Reinforcement LearningMathematical ReasoningCode GenerationAutonomous Strawberry Picking Robotic System (Robofruit)
Challenges in strawberry picking made selective harvesting robotic technology demanding. However, selective harvesting of strawberries is complicated forming a few scientific research questions. Most available solutions …