Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
Vision-Language Models like CLIP create aligned embedding spaces for text and images, making it possible for anyone to build a visual classifier by simply naming the classes they want to distinguish. However, a model that works well in one domain may fail in another, and non-expert users have no straightforward way to assess whether their chosen VLM will work on their problem. We build on prior work using text-only comparisons to evaluate how well a model works for a given natural language task, and explore approaches that also generate synthetic images relevant to that task to evaluate and refine the prediction of zero-shot accuracy. We show that generated imagery to the baseline text-only scores substantially improves the quality of these predictions. Additionally, it gives a user feedback on the kinds of images that were used to make the assessment. Experiments on standard CLIP benchmark datasets demonstrate that the image-based approach helps users predict, without any labeled examples, whether a VLM will be effective for their application.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Zero-shot object prediction using semantic scene knowledge
This work focuses on the semantic relations between scenes and objects for visual object recognition. Semantic knowledge can be a powerful source of information especially in scenarios with few or no annotated training s…
ObjectObject RecognitionPredictionSEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models
Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing.The canonical utterance is often lengthy and complex due to the compositional …
Out-of-Distribution GeneralizationSemantic ParsingSeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models
Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing. The canonical utterance is often lengthy and complex due to the compositional…
Out-of-Distribution GeneralizationSemantic ParsingA New Split for Evaluating True Zero-Shot Action Recognition
Zero-shot action recognition is the task of classifying action categories that are not available in the training set. In this setting, the standard evaluation protocol is to use existing action recognition datasets(e.g. …
Action RecognitionFew-Shot action recognitionFew Shot Action RecognitionZero-Shot Action Recognition+1On the Zero-shot Adversarial Robustness of Vision-Language Models: A Truly Zero-shot and Training-free Approach
Pre-trained Vision-Language Models (VLMs) like CLIP, have demonstrated strong zero-shot generalization capabilities. Despite their effectiveness on various downstream tasks, they remain vulnerable to adversarial samp…
Adversarial RobustnessZero-shot Generalization