Elastic Architecture Search for Efficient Language Models
As large pre-trained language models become increasingly critical to natural language understanding (NLU) tasks, their substantial computational and memory requirements have raised significant economic and environmental concerns. Addressing these challenges, this paper introduces the Elastic Language Model (ELM), a novel neural architecture search (NAS) method optimized for compact language models. ELM extends existing NAS approaches by introducing a flexible search space with efficient transformer blocks and dynamic modules for dimension and head number adjustment. These innovations enhance the efficiency and flexibility of the search process, which facilitates more thorough and effective exploration of model architectures. We also introduce novel knowledge distillation losses that preserve the unique characteristics of each block, in order to improve the discrimination between architectural choices during the search process. Experiments on masked language modeling and causal language modeling tasks demonstrate that models discovered by ELM significantly outperform existing methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingNeural Architecture SearchKnowledge DistillationSimilar Papers 제목 키워드 기반
On Elastic Language Models
Large-scale pretrained language models have achieved compelling performance in a wide range of language understanding and information retrieval tasks. Knowledge distillation offers an opportunity to compress a large lang…
Information RetrievalKnowledge DistillationLanguage ModelingLanguage Modelling+4ElasticRec: A Microservice-based Model Serving Architecture Enabling Elastic Resource Scaling for Recommendation Models
With the increasing popularity of recommendation systems (RecSys), the demand for compute resources in datacenters has surged. However, the model-wise resource allocation employed in current RecSys model serving architec…
Recommendation SystemsEnhancing Cloud-Based Large Language Model Processing with Elasticsearch and Transformer Models
Large Language Models (LLMs) are a class of generative AI models built using the Transformer network, capable of leveraging vast datasets to identify, summarize, translate, predict, and generate language. LLMs promise to…
Language ModelingLanguage ModellingLarge Language ModelStar Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control
Training a family of large language models (LLMs), either from scratch or via iterative compression, is prohibitively expensive and inefficient, requiring separate training runs for each model in the family. In this pape…
Knowledge DistillationEAT-NAS: Elastic Architecture Transfer for Accelerating Large-scale Neural Architecture Search
Neural architecture search (NAS) methods have been proposed to release human experts from tedious architecture engineering. However, most current methods are constrained in small-scale search due to the issue of computat…
Neural Architecture Search