CLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval
Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealing with limited large-scale datasets. In this paper, we introduce a CLIP-based Synergistic Knowledge Transfer (CSKT) approach for TPR. Specifically, to explore the CLIP's knowledge on input side, we first propose a Bidirectional Prompts Transferring (BPT) module constructed by text-to-image and image-to-text bidirectional prompts and coupling projections. Secondly, Dual Adapters Transferring (DAT) is designed to transfer knowledge on output side of Multi-Head Attention (MHA) in vision and language. This synergistic two-way collaborative mechanism promotes the early-stage feature fusion and efficiently exploits the existing knowledge of CLIP. CSKT outperforms the state-of-the-art approaches across three benchmark datasets when the training parameters merely account for 7.4% of the entire model, demonstrating its remarkable efficiency, effectiveness and generalization.
Code (1)
Tasks
Image to textPerson RetrievalRetrievalText based Person RetrievalText-based Person RetrievalTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning
Few-shot learning (FSL) addresses the challenge of classifying novel classes with limited training samples. While some methods leverage semantic knowledge from smaller-scale models to mitigate data scarcity, these approa…
Few-Shot LearningTF-CLIP: Learning Text-free CLIP for Video-based Person Re-Identification
Large-scale language-image pre-trained models (e.g., CLIP) have shown superior performances on many cross-modal retrieval tasks. However, the problem of transferring the knowledge learned from such models to video-based …
Cross-Modal RetrievalPerson Re-IdentificationVideo-Based Person Re-IdentificationUP-Person: Unified Parameter-Efficient Transfer Learning for Text-based Person Retrieval
Text-based Person Retrieval (TPR) as a multi-modal task, which aims to retrieve the target person from a pool of candidate images given a text description, has recently garnered considerable attention due to the progress…
Person RetrievalRetrievalText based Person RetrievalText-based Person Retrieval+1Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval
Although Contrastive Language-Image Pre-training (CLIP) exhibits strong performance across diverse vision tasks, its application to person representation learning faces two critical challenges: (i) the scarcity of large-…
Representation LearningContrastive LearningPerson RetrievalHierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning
Class-Incremental Learning (CIL) enables models to learn new classes continually while preserving past knowledge. Recently, vision-language models like CLIP offer transferable features via multi-modal pre-training, makin…
class-incremental learning