Logits DeConfusion with CLIP for Few-Shot Learning

2025-04-16 · CVPR 2025 1 · Shuo Li, Fang Liu, Zehua Hao, Xinyi Wang, Lingling Li, Xu Liu, Puhua Chen, Wenping Ma

With its powerful visual-language alignment capability, CLIP performs well in zero-shot and few-shot learning tasks. However, we found in experiments that CLIP's logits suffer from serious inter-class confusion problems in downstream tasks, and the ambiguity between categories seriously affects the accuracy. To address this challenge, we propose a novel method called Logits DeConfusion, which effectively learns and eliminates inter-class confusion in logits by combining our Multi-level Adapter Fusion (MAF) module with our Inter-Class Deconfusion (ICD) module. Our MAF extracts features from different levels and fuses them uniformly to enhance feature representation. Our ICD learnably eliminates inter-class confusion in logits with a residual structure. Experimental results show that our method can significantly improve the classification performance and alleviate the inter-class confusion problem. The code is available at https://github.com/LiShuo1001/LDC.

📄 PDF Abstract BibTeX arXiv:2504.12104

Code (1)

lishuo1001/ldc 공식 구현 pytorch

Tasks

Few-Shot Learning

Methods 이 논문이 사용한 방법론

Adapter 설명 없음

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models

2026-07-10 · Tianyou Jiang, Ziyu Zhou arxiv

Medical Vision-Language Models (VLMs) exhibit strong zero-shot performance, yet their effectiveness still declines on out-of-distribution (OOD) data due to domain shifts and class bias inherited from large-scale pretrain…

Calibrating Uncertainty for Zero-Shot Adversarial CLIP

2025-12-15 · Wenjing Lu, Zerui Tao, Yuning Qiu, Dongping Zhang 외 arxiv

CLIP delivers strong zero-shot classification but remains highly vulnerable to adversarial attacks. Prior adversarial fine-tuning work primarily matches predicted logits between clean and adversarial examples, which over…

Zero-shot Generalization Adversarial Robustness

SeMoBridge: Semantic Modality Bridge for Efficient Few-Shot Adaptation of CLIP

2025-09-30 · Christoph Timmermann, Hyunse Lee, Woojin Lee arxiv

While Contrastive Language-Image Pretraining (CLIP) excels at zero-shot tasks by aligning image and text embeddings, its performance in few-shot classification is hindered by a critical limitation: intra-modal misalignme…

Robust Calibration of Large Vision-Language Adapters

2024-07-18 · Balamurali Murugesan, Julio Silva-Rodriguez, Ismail Ben Ayed, Jose Dolz

This paper addresses the critical issue of miscalibration in CLIP-based model adaptation, particularly in the challenging scenario of out-of-distribution (OOD) samples, which has been overlooked in the existing literatur…

Prompt Learning Test-time Adaptation

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

2026-06-21 · Na Sang, Ding Ma, Rui Sang, Yuxuan Liu arxiv

Few-shot prompt learning is an effective strategy for adapting CLIP to downstream tasks, but class-only prompt optimization can overfit base-class supervision and weaken transfer to unseen classes. We propose Concept-Con…