What do we learn from inverting CLIP models?
We employ an inversion-based approach to examine CLIP models. Our examination reveals that inverting CLIP models results in the generation of images that exhibit semantic alignment with the specified target prompts. We leverage these inverted images to gain insights into various aspects of CLIP models, such as their ability to blend concepts and inclusion of gender biases. We notably observe instances of NSFW (Not Safe For Work) images during model inversion. This phenomenon occurs even for semantically innocuous prompts, like "a beautiful landscape," as well as for prompts involving the names of celebrities.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
Contrastive Language-Image Pre-training (CLIP) has become a foundation model and has been applied to various vision and multimodal tasks. However, recent works indicate that CLIP falls short in distinguishing detailed di…
Large Language ModelMultimodal Large Language ModelCLIP-FTI: Fine-Grained Face Template Inversion via CLIP-Driven Attribute Conditioning
Face recognition systems store face templates for efficient matching. Once leaked, these templates pose a threat: inverting them can yield photorealistic surrogates that compromise privacy and enable impersonation. Altho…
Face RecognitionWhat Vision-Language Models `See' when they See Scenes
Images can be described in terms of the objects they contain, or in terms of the types of scene or place that they instantiate. In this paper we address to what extent pretrained Vision and Language models can learn to a…
ObjectEffectively Leveraging CLIP for Generating Situational Summaries of Images and Videos
Situation recognition refers to the ability of an agent to identify and understand various situations or contexts based on available information and sensory inputs. It involves the cognitive process of interpreting data …
Semantic Role LabelingVideo CaptioningWhat Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective
Vision-Language Models (VLMs) such as CLIP have become a standard backbone for open-vocabulary recognition, yet their zero-shot predictions remain vulnerable to distribution shifts encountered at deployment. Test-Time Ad…
Test-time Adaptation