CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
CLIP has shown a remarkable zero-shot capability on a wide range of vision tasks. Previously, CLIP is only regarded as a powerful visual encoder. However, after being pre-trained by language supervision from a large amount of image-caption pairs, CLIP itself should also have acquired some few-shot abilities for vision-language tasks. In this work, we empirically show that CLIP can be a strong vision-language few-shot learner by leveraging the power of language. We first evaluate CLIP's zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task. Then we propose a parameter-efficient fine-tuning strategy to boost the few-shot performance on the vqa task. We achieve competitive zero/few-shot results on the visual question answering and visual entailment tasks without introducing any additional pre-training procedure.
Code (0)
등록된 구현이 없습니다.
Tasks
parameter-efficient fine-tuningQuestion AnsweringVisual EntailmentVisual Question AnsweringVisual Question Answering (VQA)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Generalized Zero-Shot Learners for Open-Domain Image Geolocalization
Image geolocalization is the challenging task of predicting the geographic coordinates of origin for a given photo. It is an unsolved problem relying on the ability to combine visual clues with general knowledge about th…
Generalized Zero-Shot LearningMeta-LearningPhoto geolocation estimationZero-Shot LearningDiagnosing Ensemble Few-Shot Classifiers
The base learners and labeled samples (shots) in an ensemble few-shot classifier greatly affect the model performance. When the performance is not satisfactory, it is usually difficult to understand the underlying causes…
Bootstrap Fine-Grained Vision-Language Alignment for Unified Zero-Shot Anomaly Localization
Contrastive Language-Image Pre-training (CLIP) models have shown promising performance on zero-shot visual recognition tasks by learning visual representations under natural language supervision. Recent studies attempt t…
Anomaly DetectionAnomaly LocalizationTest-time Adaptationzero-shot anomaly detectionPrompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown promising few-shot performance benefited fr…
Few-Shot LearningRepresentation LearningMaking Pretrained Language Models Good Long-tailed Learners
Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hypothesis that prompt-tuning is also a…
Classification