Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision
Parameter-efficient fine-tuning (PEFT) has become increasingly important as foundation models continue to grow in both popularity and size. Adapter has been particularly well-received due to their potential for parameter reduction and adaptability across diverse tasks. However, striking a balance between high efficiency and robust generalization across tasks remains a challenge for adapter-based methods. We analyze existing methods and find that: 1) parameter sharing is the key to reducing redundancy; 2) more tunable parameters, dynamic allocation, and block-specific design are keys to improving performance. Unfortunately, no previous work considers all these factors. Inspired by this insight, we introduce a novel framework named Adapter-X. First, a Sharing Mixture of Adapters (SMoA) module is proposed to fulfill token-level dynamic allocation, increased tunable parameters, and inter-block sharing at the same time. Second, some block-specific designs like Prompt Generator (PG) are introduced to further enhance the ability of adaptation. Extensive experiments across 2D image and 3D point cloud modalities demonstrate that Adapter-X represents a significant milestone as it is the first to outperform full fine-tuning in both 2D image and 3D point cloud modalities with significantly fewer parameters, i.e., only 0.20% and 1.88% of original trainable parameters for 2D and 3D classification tasks. Our code will be publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
3D Classificationparameter-efficient fine-tuningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification
With excellent generalization ability, self-supervised speech models have shown impressive performance on various downstream speech tasks in the pre-training and fine-tuning paradigm. However, as the growing size of pre-…
Speaker VerificationTransfer LearningLLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific da…
Arithmetic ReasoningLanguage ModellingAdapterEM: Pre-trained Language Model Adaptation for Generalized Entity Matching using Adapter-tuning
Entity Matching (EM) involves identifying different data representations referring to the same entity from multiple data sources and is typically formulated as a binary classification problem. It is a challenging problem…
Binary ClassificationData IntegrationLanguage ModelingLanguage Modelling+1AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs
Fine-tuning pre-trained models has recently yielded remarkable performance gains in graph neural networks (GNNs). In addition to pre-training techniques, inspired by the latest work in the natural language fields, more r…
Generalization Boundsparameter-efficient fine-tuningConv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets
While parameter efficient tuning (PET) methods have shown great potential with transformer architecture on Natural Language Processing (NLP) tasks, their effectiveness with large-scale ConvNets is still under-studied on …
Transfer Learning