Unsupervised Prompt Learning for Vision-Language Models
Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.
Code (1)
Tasks
Prompt EngineeringPrompt LearningTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Large (Vision) Language Models are Unsupervised In-Context Learners
Recent advances in large language and vision-language models have enabled zero-shot inference, allowing models to solve new tasks without task-specific training. Various adaptation techniques such as prompt engineering, …
GSM8KIn-Context LearningMathPrompt EngineeringUnsupervised Pre-training with Language-Vision Prompts for Low-Data Instance Segmentation
In recent times, following the paradigm of DETR (DEtection TRansformer), query-based end-to-end instance segmentation (QEIS) methods have exhibited superior performance compared to CNN-based models, particularly when tra…
Instance SegmentationSemantic SegmentationUnsupervised Pre-trainingUP-DP: Unsupervised Prompt Learning for Data Pre-Selection with Vision-Language Models
In this study, we investigate the task of data pre-selection, which aims to select instances for labeling from an unlabeled dataset through a single pass, thereby optimizing performance for undefined downstream tasks wit…
Language ModellingPrompt LearningExploring Low-Resource Medical Image Classification with Weakly Supervised Prompt Learning
Most advances in medical image recognition supporting clinical auxiliary diagnosis meet challenges due to the low-resource situation in the medical field, where annotations are highly expensive and professional. This low…
Few-Shot Learningimage-classificationImage ClassificationLanguage Modeling+4In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model
Current pre-trained vision-language models, such as CLIP, have demonstrated remarkable zero-shot generalization capabilities across various downstream tasks. However, their performance significantly degrades when test in…
In-Context LearningLanguage ModelingLanguage ModellingPrompt Learning+1