SMoA: Sparse Mixture of Adapters to Mitigate Multiple Dataset Biases
Recent studies reveal that various biases exist in different NLP tasks, and over-reliance on biases results in models' poor generalization ability and low adversarial robustness. To mitigate datasets biases, previous works propose lots of debiasing techniques to tackle specific biases, which perform well on respective adversarial sets but fail to mitigate other biases. In this paper, we propose a new debiasing method Sparse Mixture-of-Adapters (SMoA), which can mitigate multiple dataset biases effectively and efficiently. Experiments on Natural Language Inference and Paraphrase Identification tasks demonstrate that SMoA outperforms full-finetuning, adapter tuning baselines, and prior strong debiasing methods. Further analysis indicates the interpretability of SMoA that sub-adapter can capture specific pattern from the training data and specialize to handle specific bias.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial RobustnessNatural Language InferenceParaphrase IdentificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
While multi-agent systems have been shown to significantly enhance the performance of Large Language Models (LLMs) across various tasks and applications, the dense interaction between scaling agents potentially hampers t…
DiversityFairnessExploring Training on Heterogeneous Data with Mixture of Low-rank Adapters
Training a unified model to take multiple targets into account is a trend towards artificial general intelligence. However, how to efficiently mitigate the training conflicts among heterogeneous data collected from diffe…
MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning
With the rapid growth in the scale of pre-trained foundation models, parameter-efficient fine-tuning techniques have gained significant attention, among which Adapter Tuning is the most widely used. Despite achieving eff…
parameter-efficient fine-tuningSMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning
As the number of model parameters increases, parameter-efficient fine-tuning (PEFT) has become the go-to choice for tailoring pre-trained large language models. Low-rank Adaptation (LoRA) uses a low-rank update method to…
parameter-efficient fine-tuningLittle By Little: Continual Learning via Self-Activated Sparse Mixture-of-Rank Adaptive Learning
Continual learning (CL) with large pre-trained models is challenged by catastrophic forgetting and task interference. Existing LoRA-based Mixture-of-Experts (MoE) approaches mitigate forgetting by assigning and freezing …
Continual LearningMixture-of-Experts