Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks
Transformer-based NLP models are powerful but have high computational costs that limit deployment. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks where multiple outputs are required for a single shared input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding and increasing the operational intensity (ratio of numbers of arithmetic operation to memory access) of decoding process by sharing the input key-value cache. We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance.
Code (1)
Tasks
DecoderDialogue State TrackingQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models
Encoder-decoder transformer models have achieved great success on various vision-language (VL) tasks, but they suffer from high inference latency. Typically, the decoder takes up most of the latency because of the auto-r…
DecoderBalancing Cost and Benefit with Tied-Multi Transformers
We propose and evaluate a novel procedure for training multiple Transformers with tied parameters which compresses multiple models into one enabling the dynamic choice of the number of encoder and decoder layers during d…
DecoderKnowledge DistillationMachine TranslationModel Compression+1Rethinking Encoder-Decoder Flow Through Shared Structures
Dense prediction tasks have enjoyed a growing complexity of encoder architectures, decoders, however, have remained largely the same. They rely on individual blocks decoding intermediate feature maps sequentially. We int…
DecoderDepth EstimationGraph-Aware Transformer: Is Attention All Graphs Need?
Graphs are the natural data structure to represent relational and structural information in many domains. To cover the broad range of graph-data applications including graph classification as well as graph generation, it…
AllDecoderGraph ClassificationGraph GenerationIs Encoder-Decoder Redundant for Neural Machine Translation?
Encoder-decoder architecture is widely adopted for sequence-to-sequence modeling tasks. For machine translation, despite the evolution from long short-term memory networks to Transformer networks, plus the introduction a…
DecoderLanguage ModelingLanguage ModellingMachine Translation+2