DetailCLIP: Injecting Image Details into CLIP's Feature Space
Although CLIP-like Visual Language Models provide a functional joint feature space for image and text, due to the limitation of the CILP-like model's image input size (e.g., 224), subtle details are lost in the feature representation if we input high-resolution images (e.g., 2240). In this work, we introduce an efficient framework that can produce a single feature representation for a high-resolution image that injects image details and shares the same semantic space as the original CLIP. In the framework, we train a feature fusing model based on CLIP features extracted from a carefully designed image patch method that can cover objects of any scale, weakly supervised by image-agnostic class prompted queries. We validate our framework by retrieving images from class prompted queries on the real world and synthetic datasets, showing significant performance improvement on these tasks. Furthermore, to fully demonstrate our framework's detail retrieval ability, we construct a CLEVR-like synthetic dataset called CLVER-DS, which is fully annotated and has a controllable object scale.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks
In this paper, we introduce DetailCLIP: A Detail-Oriented CLIP to address the limitations of contrastive learning-based vision-language models, particularly CLIP, in handling detail-oriented and fine-grained tasks like s…
Contrastive LearningImage ReconstructionLanguage ModelingLanguage Modelling+2BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP
Contrastive Vision-Language Pre-training, known as CLIP, has shown promising effectiveness in addressing downstream image recognition tasks. However, recent works revealed that the CLIP model can be implanted with a down…
Prompt LearningC2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection
This work focuses on AIGC detection to develop universal detectors capable of identifying various types of forgery images. Recent studies have found large pre-trained models, such as CLIP, are effective for generalizable…
DeepFake DetectionFace SwappingLearning Joint ID-Textual Representation for ID-Preserving Image Synthesis
We propose a novel framework for ID-preserving generation using a multi-modal encoding strategy rather than injecting identity features via adapters into pre-trained models. Our method treats identity and text as a unifi…
Image GenerationExploring Intra-Class Variation Factors With Learnable Cluster Prompts for Semi-Supervised Image Synthesis
Semi-supervised class-conditional image synthesis is typically performed by inferring and injecting class labels into a conditional Generative Adversarial Network (GAN). The supervision in the form of class identity …
Conditional Image GenerationGenerative Adversarial NetworkImage Generation