Evolving Subnetwork Training for Large Language Models

2024-06-11 · Hanqi Li, Lu Chen, Da Ma, Zijian Wu, Su Zhu, Kai Yu

Large language models have ushered in a new era of artificial intelligence research. However, their substantial training costs hinder further development and widespread adoption. In this paper, inspired by the redundancy in the parameters of large language models, we propose a novel training paradigm: Evolving Subnetwork Training (EST). EST samples subnetworks from the layers of the large language model and from commonly used modules within each layer, Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP). By gradually increasing the size of the subnetworks during the training process, EST can save the cost of training. We apply EST to train GPT2 model and TinyLlama model, resulting in 26.7\% FLOPs saving for GPT2 and 25.0\% for TinyLlama without an increase in loss on the pre-training dataset. Moreover, EST leads to performance improvements in downstream tasks, indicating that it benefits generalization. Additionally, we provide intuitive theoretical studies based on training dynamics and Dropout theory to ensure the feasibility of EST. Our code is available at https://github.com/OpenDFM/EST.

📄 PDF Abstract BibTeX arXiv:2406.06962

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage ModellingLarge Language Model

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training

2022-05-30 · Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang 외

Recent works on sparse neural network training (sparse training) have shown that a compelling trade-off between performance and efficiency can be achieved by training intrinsically sparse neural networks from scratch. Ex…

Emergent Specialization: Rare Token Neurons in Language Models

2025-05-19 · Jing Liu, Haozheng Wang, Yueheng Li

Large language models struggle with representing and generating rare tokens despite their importance in specialized domains. In this study, we identify neuron structures with exceptionally strong influence on language mo…

Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks

2023-11-14 · Rochelle Choenni, Ekaterina Shutova, Dan Garrette

Recent work has proposed explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as a means of better guiding cross-lingual sharing. In this work, we inve…

Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks

2022-10-31 · Rochelle Choenni, Dan Garrette, Ekaterina Shutova

Large multilingual language models typically share their parameters across all languages, which enables cross-lingual task transfer, but learning can also be hindered when training updates from different languages are in…

Cross-Lingual Transfer Meta-Learning

The Lottery Ticket Hypothesis for Pre-trained BERT Networks

2020-07-23 · NeurIPS 2020 12 · Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu 외

In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends are emerging in other areas of deep lear…

Language Modeling Language Modelling Masked Language Modeling