BART-light: One Decoder Layer Is Enough
BART (Lewis et al., 2020), an encoder-decoder transformer language model (LM), has reached state-of-the-art results on several tasks in natural language generation and understanding. Similar to other pretrained encoder-decoder LMs, it uses the same number of hidden layers in the encoder and the decoder. In this paper, we show that one can easily remove all but one or two decoder layers for text generation tasks and even remove the whole decoder for classification tasks, with little to no compromises on performance. Our study presents that a shallow decoder is sufficient for most tasks when a deep encoder is used.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLanguage ModelingLanguage ModellingText GenerationSimilar Papers 제목 키워드 기반
Stylistic MR-to-Text Generation Using Pre-trained Language Models
We explore the ability of pre-trained language models BART, an encoder-decoder model, GPT2 and GPT-Neo, both decoder-only models for generating sentences from structured MR tags as input. We observe best results on sever…
DecoderPOSSentenceText GenerationParallel Refinements for Lexically Constrained Text Generation with BART
Lexically constrained text generation aims to control the generated text by incorporating some pre-specified keywords into the output. Previous work injects lexical constraints into the output by controlling the decoding…
DecoderDiversitySentenceText GenerationLearning Cluster Patterns for Abstractive Summarization
Nowadays, pre-trained sequence-to-sequence models such as BERTSUM and BART have shown state-of-the-art results in abstractive summarization. In these models, during fine-tuning, the encoder transforms sentences to contex…
Abstractive Text SummarizationDecoderInvestigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstracts
Biomedical literature often uses complex language and inaccessible professional terminologies. That is why simplification plays an important role in improving public health literacy. Applying Natural Language Processing …
DecoderText SimplificationAraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization
Like most natural language understanding and generation tasks, state-of-the-art models for summarization are transformer-based sequence-to-sequence architectures that are pretrained on large corpora. While most existing …
Abstractive Text SummarizationDecoderNatural Language Understanding