Adapter-state Sharing CLIP for Parameter-efficient Multimodal Sarcasm Detection
The growing prevalence of multimodal image-text sarcasm on social media poses challenges for opinion mining systems. Existing approaches rely on full fine-tuning of large models, making them unsuitable to adapt under resource-constrained settings. While recent parameter-efficient fine-tuning (PEFT) methods offer promise, their off-the-shelf use underperforms on complex tasks like sarcasm detection. We propose AdS-CLIP (Adapter-state Sharing in CLIP), a lightweight framework built on CLIP that inserts adapters only in the upper layers to preserve low-level unimodal representations in the lower layers and introduces a novel adapter-state sharing mechanism, where textual adapters guide visual ones to promote efficient cross-modal learning in the upper layers. Experiments on two public benchmarks demonstrate that AdS-CLIP not only outperforms standard PEFT methods but also existing multimodal baselines with significantly fewer trainable parameters.
Code (0)
등록된 구현이 없습니다.
Tasks
parameter-efficient fine-tuningSarcasm DetectionOpinion MiningSimilar Papers 제목 키워드 기반
MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval
State-of-the-art video-text retrieval (VTR) methods typically involve fully fine-tuning a pre-trained model (e.g. CLIP) on specific datasets. However, this can result in significant storage costs in practical application…
RetrievalText RetrievalTransfer LearningVideo-Text RetrievalRMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models
Pre-trained Vision-Language Models (VLMs), \textit{e.g.} CLIP, have become essential tools in multimodal transfer learning. However, fine-tuning VLMs in few-shot scenarios poses significant challenges in balancing task-s…
parameter-efficient fine-tuningDomain GeneralizationGeneral KnowledgeTransfer LearningMA-CLIP: Towards Modality-Agnostic Contrastive Language-Image Pre-training
Large-scale multimodal contrastive pretraining has demonstrated great utility to support high performance in a range of downstream tasks by mapping multiple modalities into a shared embedding space. Typically, this has e…
CapS-Adapter: Caption-based MultiModal Adapter in Zero-Shot Classification
Recent advances in vision-language foundational models, such as CLIP, have demonstrated significant strides in zero-shot classification. However, the extensive parameterization of models like CLIP necessitates a resource…
zero-shot-classificationZero-Shot LearningMIP: CLIP-based Image Reconstruction from PEFT Gradients
Contrastive Language-Image Pre-training (CLIP) model, as an effective pre-trained multimodal neural network, has been widely used in distributed machine learning tasks, especially Federated Learning (FL). Typically, CLIP…
Federated LearningImage Reconstructionparameter-efficient fine-tuningReconstruction Attack