Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. Common neural network compression techniques such as knowledge distillation or quantization are limited to static compression where the compression ratio is fixed. In this paper, we introduce Modular Transformers, a modularized encoder-decoder framework for flexible sequence-to-sequence model compression. Modular Transformers train modularized layers that have the same function of two or more consecutive layers in the original model via module replacing and knowledge distillation. After training, the modularized layers can be flexibly assembled into sequence-to-sequence models that meet different performance-efficiency trade-offs. Experimental results show that after a single training phase, by simply varying the assembling strategy, Modular Transformers can achieve flexible compression ratios from 1.1x to 6x with little to moderate relative performance drop.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderKnowledge DistillationModel CompressionNeural Network CompressionQuantizationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Modularized Transfomer-based Ranking Framework
Recent innovations in Transformer-based ranking models have advanced the state-of-the-art in information retrieval. However, these Transformers are computationally expensive, and their opaque hidden states make it hard t…
Information RetrievalRetrievalMICSim: A Modular Simulator for Mixed-signal Compute-in-Memory based AI Accelerator
This work introduces MICSim, an open-source, pre-circuit simulator designed for early-stage evaluation of chip-level software performance and hardware overhead of mixed-signal compute-in-memory (CIM) accelerators. MICSim…
QuantizationModular addition without black-boxes: Compressing explanations of MLPs that compute numerical integration
The goal of mechanistic interpretability is discovering simpler, low-rank algorithms implemented by models. While we can compress activations into features, compressing nonlinear feature-maps -- like MLP layers -- is an …
Numerical IntegrationVDT: General-purpose Video Diffusion Transformers via Mask Modeling
This work introduces Video Diffusion Transformer (VDT), which pioneers the use of transformers in diffusion-based video generation. It features transformer blocks with modularized temporal and spatial attention modules t…
Autonomous DrivingVideo GenerationVideo PredictionExACT: An End-to-End Autonomous Excavator System Using Action Chunking With Transformers
Excavators are crucial for diverse tasks such as construction and mining, while autonomous excavator systems enhance safety and efficiency, address labor shortages, and improve human working conditions. Different from th…
ChunkingImitation Learning