EncT5: Fine-tuning T5 Encoder for Discriminative Tasks
Encoder-decoder transformer architectures have become popular recently with the advent of T5 models. While they demonstrate impressive performance on benchmarks such as GLUE (Wang et al., 2019), it is not clearly evident if the proposed encoder-decoder architecture is the most efficient for fine-tuning on downstream discriminative tasks. In this work, we study fine-tuning pre-trained encoderdecoder models such as T5. Particularly, we propose EncT5 as a way to efficiently finetune pre-trained encoder-decoder T5 models for classification and regression tasks by using only the encoder layers. Our experimental results show that EncT5 with less than half of the parameters of T5 performs similarly to T5 models on GLUE benchmark. We believe our proposed approach can be easily applied to any pre-trained encoder-decoder model.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
EncT5: A Framework for Fine-tuning T5 as Non-autoregressive Models
Pre-trained encoder-decoder transformer architectures have become increasingly popular recently with the advent of T5 models. T5 has also become more favorable over other architectures like BERT due to the amount of data…
DecoderLanguage ModellingMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATION+2A New Probabilistic V-Net Model with Hierarchical Spatial Feature Transform for Efficient Abdominal Multi-Organ Segmentation
Accurate and robust abdominal multi-organ segmentation from CT imaging of different modalities is a challenging task due to complex inter- and intra-organ shape and appearance variations among abdominal organs. In this p…
DecoderOrgan SegmentationSegmentationAbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?
With the unprecedented developments in deep learning, automatic segmentation of main abdominal organs seems to be a solved problem as state-of-the-art (SOTA) methods have achieved comparable results with inter-rater vari…
Continual LearningOrgan SegmentationPancreas SegmentationSegmentationEfficient and Versatile Robust Fine-Tuning of Zero-shot Models
Large-scale image-text pre-trained models enable zero-shot classification and provide consistent accuracy across various data distributions. Nonetheless, optimizing these models in downstream tasks typically requires fin…
Cross-Modal Retrievalzero-shot-classificationZero-Shot LearningPre-Calc: Learning to Use the Calculator Improves Numeracy in Language Models
Quantitative and numerical comprehension in language is an important task in many fields like education and finance, but still remains a challenging task for language models. While tool and calculator usage has shown to …
DecoderMathematical Reasoning