Prompt Tuning for Zero-shot Compositional Learning
Open World Compositional Zero-Shot Learning (OW-CZSL) is known to be an extremely challenging task, which aims to recognize unseen compositions formed from seen attributes and objects without any prior assumption of the output space. In order to achieve this goal, a model has to be "smart" and "knowledgeable". To be smart, a model should be good at reasoning the interactions between attributes and objects from the seen compositions. While "knowledgeable" means the model owns "common sense" to the open world that can "foresee" some features of the unseen compositions. Most previous work focuses on the "smart" part, while few of them provided an effective solution to achieve the "knowledgeable" goal. In this paper, we proposed a framework named Multi-Modal Prompt Tuning (MMPT) to inherit the "knowledgeable" property from the large pre-trained vision-language model. Extensive experiments show that our proposed MMPT obtains new state-of-the-art results in OW-CZSL task. On the UT-Zappos dataset, MMPT pushes the AUC score to $29.8$, while the previous best score is $26.5$. On the more challenging MIT-States dataset, the AUC score of MMPT is 1.5 times better than the current state-of-the-art.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningCompositional Zero-Shot LearningLanguage ModellingZero-Shot LearningSimilar Papers 제목 키워드 기반
Plug-and-Play Emotion Graphs for Compositional Prompting in Zero-Shot Speech Emotion Recognition
Large audio-language models (LALMs) exhibit strong zero-shot performance across speech tasks but struggle with speech emotion recognition (SER) due to weak paralinguistic modeling and limited cross-modal reasoning. We pr…
Speech Emotion RecognitionSEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models
Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing.The canonical utterance is often lengthy and complex due to the compositional …
Out-of-Distribution GeneralizationSemantic ParsingSeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models
Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing. The canonical utterance is often lengthy and complex due to the compositional…
Out-of-Distribution GeneralizationSemantic ParsingPrompting Language-Informed Distribution for Compositional Zero-Shot Learning
Compositional zero-shot learning (CZSL) task aims to recognize unseen compositional visual concepts, e.g., sliced tomatoes, where the model is learned only from the seen compositions, e.g., sliced potatoes and red tomato…
Compositional Zero-Shot LearningInformativenessZero-shot GeneralizationZero-Shot LearningIntCoOp: Interpretability-Aware Vision-Language Prompt Tuning
Image-text contrastive models such as CLIP learn transferable and robust representations for zero-shot transfer to a variety of downstream tasks. However, to obtain strong downstream performances, prompts need to be care…
AttributeFew-Shot LearningPrompt Engineering