Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition
Large-scale pre-trained models have achieved remarkable success in language and image tasks, leading an increasing number of studies to explore the application of pre-trained image models, such as CLIP, in the domain of few-shot action recognition (FSAR). However, current methods generally suffer from several problems: 1) Direct fine-tuning often undermines the generalization capability of the pre-trained model; 2) The exploration of task-specific information is insufficient in the visual tasks; 3) The semantic order information is typically overlooked during text modeling; 4) Existing cross-modal alignment techniques ignore the temporal coupling of multimodal information. To address these, we propose Task-Adapter++, a parameter-efficient dual adaptation method for both image and text encoders. Specifically, to make full use of the variations across different few-shot learning tasks, we design a task-specific adaptation for the image encoder so that the most discriminative information can be well noticed during feature extraction. Furthermore, we leverage large language models (LLMs) to generate detailed sequential sub-action descriptions for each action class, and introduce semantic order adapters into the text encoder to effectively model the sequential relationships between these sub-actions. Finally, we develop an innovative fine-grained cross-modal alignment strategy that actively maps visual features to reside in the same temporal stage as semantic descriptions. Extensive experiments fully demonstrate the effectiveness and superiority of the proposed method, which achieves state-of-the-art performance on 5 benchmarks consistently. The code is open-sourced at https://github.com/Jaulin-Bage/Task-Adapter-pp.
Code (1)
Tasks
Action Recognitioncross-modal alignmentFew-Shot action recognitionFew Shot Action RecognitionFew-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Efficient Domain Adaptation of Sentence Embeddings Using Adapters
Sentence embeddings enable us to capture the semantic similarity of short texts. Most sentence embedding models are trained for general semantic textual similarity tasks. Therefore, to use sentence embeddings in a partic…
Domain AdaptationSemantic SimilaritySemantic Textual SimilaritySentence+4ReLoRA: Knowledge-Reusing Adaptation for Fast Rollout of Evolving LLM Services
Large Language Models (LLMs) are increasingly deployed as continuously evolving services, where frequent base-model updates may invalidate previously deployed task-specific Low-Rank Adaptation (LoRA) adapters. For servic…
Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition
Existing works in few-shot action recognition mostly fine-tune a pre-trained image model and design sophisticated temporal alignment modules at feature level. However, simply fully fine-tuning the pre-trained model could…
Action RecognitionFew-Shot action recognitionFew Shot Action RecognitionHierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models
Parameter efficient adaptation methods have become a key mechanism to train large pre-trained models for downstream tasks. However, their per-task parameter overhead is considered still high when the number of downstream…
Automatic Speech Recognitionspeech-recognitionSpeech RecognitionRMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models
Pre-trained Vision-Language Models (VLMs), \textit{e.g.} CLIP, have become essential tools in multimodal transfer learning. However, fine-tuning VLMs in few-shot scenarios poses significant challenges in balancing task-s…
parameter-efficient fine-tuningDomain GeneralizationGeneral KnowledgeTransfer Learning