Transferring Core Knowledge via Learngenes

2024-01-16 · Fu Feng, Jing Wang, Xin Geng

The pre-training paradigm fine-tunes the models trained on large-scale datasets to downstream tasks with enhanced performance. It transfers all knowledge to downstream tasks without discriminating which part is necessary or unnecessary, which may lead to negative transfer. In comparison, knowledge transfer in nature is much more efficient. When passing genetic information to descendants, ancestors encode only the essential knowledge into genes, which act as the medium. Inspired by that, we adopt a recent concept called ``learngene'' and refine its structures by mimicking the structures of natural genes. We propose the Genetic Transfer Learning (GTL) -- a framework to copy the evolutionary process of organisms into neural networks. GTL trains a population of networks, selects superior learngenes by tournaments, performs learngene mutations, and passes the learngenes to next generations. Finally, we successfully extract the learngenes of VGG11 and ResNet12. We show that the learngenes bring the descendant networks instincts and strong learning ability: with 20% parameters, the learngenes bring 12% and 16% improvements of accuracy on CIFAR-FS and miniImageNet. Besides, the learngenes have the scalability and adaptability on the downstream structure of networks and datasets. Overall, we offer a novel insight that transferring core knowledge via learngenes may be sufficient and efficient for neural networks.

📄 PDF Abstract BibTeX arXiv:2401.08139

Code (0)

등록된 구현이 없습니다.

Tasks

Transfer Learning

KIND: Knowledge Integration and Diversion for Training Decomposable Models

2024-08-14 · Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang 외

Pre-trained models have become the preferred backbone due to the increasing complexity of model parameters. However, traditional pre-trained models often face deployment challenges due to their fixed sizes, and are prone…

parameter-efficient fine-tuning

Genes in Intelligent Agents

2023-06-17 · Fu Feng, Jing Wang, Xu Yang, Xin Geng

The genes in nature give the lives on earth the current biological intelligence through transmission and accumulation over billions of years. Inspired by the biological intelligence, artificial intelligence (AI) has devo…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

WAVE: Weight Template for Adaptive Initialization of Variable-sized Models

2024-06-25 · Fu Feng, Yucheng Xie, Jing Wang, Xin Geng

The expansion of model parameters underscores the significance of pre-trained models; however, the constraints encountered during model deployment necessitate models of variable sizes. Consequently, the traditional pre-t…

Knowledge Distillation Transfer Learning

Learngene Search Across Multiple Datasets for Building Variable-Sized Models

2026-05-06 · Boyu Shi, Junbo Zhou, Chang Liu, Xu Yang 외 arxiv

Deep learning methods are widely used under diverse resource constraints, resulting in models of varying sizes, such as the Vision Transformer (ViT) series. Deploying these models typically requires costly pretraining an…

Knowledge Diversion for Efficient Morphology Control and Policy Transfer

2025-12-10 · Fu Feng, Ruixiao Shi, Yucheng Xie, Jianlu Shen 외 arxiv

Universal morphology control aims to learn a universal policy that generalizes across heterogeneous agent morphologies, with Transformer-based controllers emerging as a popular choice. However, such architectures incur s…