Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models

2023-08-22 · ICCV 2023 1 · Baoshuo Kan, Teng Wang, Wenpeng Lu, XianTong Zhen, Weili Guan, Feng Zheng

Pre-trained vision-language models, e.g., CLIP, working with manually designed prompts have demonstrated great capacity of transfer learning. Recently, learnable prompts achieve state-of-the-art performance, which however are prone to overfit to seen classes, failing to generalize to unseen classes. In this paper, we propose a Knowledge-Aware Prompt Tuning (KAPT) framework for vision-language models. Our approach takes inspiration from human intelligence in which external knowledge is usually incorporated into recognizing novel categories of objects. Specifically, we design two complementary types of knowledge-aware prompts for the text encoder to leverage the distinctive characteristics of category-related external knowledge. The discrete prompt extracts the key information from descriptions of an object category, and the learned continuous prompt captures overall contexts. We further design an adaptation head for the visual encoder to aggregate salient attentive visual cues, which establishes discriminative and task-aware visual representations. We conduct extensive experiments on 11 widely-used benchmark datasets and the results verify the effectiveness in few-shot image classification, especially in generalizing to unseen categories. Compared with the state-of-the-art CoCoOp method, KAPT exhibits favorable performance and achieves an absolute gain of 3.22% on new classes and 2.57% in terms of harmonic mean.

📄 PDF Abstract BibTeX arXiv:2308.11186

Code (0)

등록된 구현이 없습니다.

Tasks

Few-Shot Image Classification image-classification Image Classification Transfer Learning

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

Learning Generalizable Prompt for CLIP with Class Similarity Knowledge

2025-02-17 · Sehun Jung, Hyang-Won Lee

In vision-language models (VLMs), prompt tuning has shown its effectiveness in adapting models to downstream tasks. However, learned prompts struggle to generalize to unseen classes, as they tend to overfit to the classe…

Distribution-Aware Prompt Tuning for Vision-Language Models

2023-09-06 · ICCV 2023 1 · Eulrang Cho, Jooyeon Kim, Hyunwoo J. Kim

Pre-trained vision-language models (VLMs) have shown impressive performance on various downstream tasks by utilizing knowledge learned from large data. In general, the performance of VLMs on target tasks can be further i…

Visual Instance-aware Prompt Tuning

2025-07-10 · Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang 외 arxiv

Visual Prompt Tuning (VPT) has emerged as a parameter-efficient fine-tuning paradigm for vision transformers, with conventional approaches utilizing dataset-level prompts that remain the same across all input instances. …

parameter-efficient fine-tuning Visual Prompt Tuning

CAPT: Class-Aware Prompt Tuning for Federated Long-Tailed Learning with Vision-Language Model

2025-03-10 · Shihao Hou, Xinyi Shang, Shreyank N Gowda, Yang Lu 외

Effectively handling the co-occurrence of non-IID data and long-tailed distributions remains a critical challenge in federated learning. While fine-tuning vision-language models (VLMs) like CLIP has shown to be promising…

Federated Learning Language Modeling Language Modelling Prompt Learning

Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers

2025-10-29 · M Yashwanth, Sharannya Ghosh, Aditay Tripathi, Anirban Chakraborty arxiv

Visual Prompt Tuning (VPT) of pre-trained Vision Transformers (ViTs) has proven highly effective as a parameter-efficient fine-tuning technique for adapting large models to downstream tasks with limited data. Its paramet…

parameter-efficient fine-tuning Visual Prompt Tuning Federated Learning