Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks
Large pre-trained language models (PLMs) have demonstrated superior performance in industrial applications. Recent studies have explored parameter-efficient PLM tuning, which only updates a small amount of task-specific parameters while achieving both high efficiency and comparable performance against standard fine-tuning. However, all these methods ignore the inefficiency problem caused by the task-specific output layers, which is inflexible for us to re-use PLMs and introduces non-negligible parameters. In this work, we focus on the text classification task and propose plugin-tuning, a framework that further improves the efficiency of existing parameter-efficient methods with a unified classifier. Specifically, we re-formulate both token and sentence classification tasks into a unified language modeling task, and map label spaces of different tasks into the same vocabulary space. In this way, we can directly re-use the language modeling heads of PLMs, avoiding introducing extra parameters for different tasks. We conduct experiments on six classification benchmarks. The experimental results show that plugin-tuning can achieve comparable performance against fine-tuned PLMs, while further saving around 50% parameters on top of other parameter-efficient methods.
Code (1)
Tasks
ClassificationLanguage ModelingLanguage ModellingSentenceSentence Classificationtext-classificationText ClassificationSimilar Papers 제목 키워드 기반
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited. However, diff…
DiversityLanguage ModelingLanguage ModellingModel SelectionUniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited. However, diff…
DiversityLanguage ModelingLanguage ModellingModel SelectionUni-LoRA: One Vector is All You Need
Low-Rank Adaptation (LoRA) has become the de facto parameter-efficient fine-tuning (PEFT) method for large language models (LLMs) by constraining weight updates to low-rank matrices. Recent works such as Tied-LoRA, VeRA,…
AllMathematical Reasoningparameter-efficient fine-tuningTowards a Unified View on Visual Parameter-Efficient Transfer Learning
Parameter efficient transfer learning (PETL) aims at making good use of the representation knowledge in the pre-trained large models by fine-tuning a small number of parameters. Recently, taking inspiration from the natu…
Action RecognitionImage ClassificationTransfer LearningVideo RecognitionParameter-Efficient Tuning on Layer Normalization for Pre-trained Language Models
Conventional fine-tuning encounters increasing difficulties given the size of current Pre-trained Language Models, which makes parameter-efficient tuning become the focal point of frontier research. Previous methods in t…