Semantic Compositions Enhance Vision-Language Contrastive Learning

2024-07-01 · Maxwell Aladago, Lorenzo Torresani, Soroush Vosoughi

In the field of vision-language contrastive learning, models such as CLIP capitalize on matched image-caption pairs as positive examples and leverage within-batch non-matching pairs as negatives. This approach has led to remarkable outcomes in zero-shot image classification, cross-modal retrieval, and linear evaluation tasks. We show that the zero-shot classification and retrieval capabilities of CLIP-like models can be improved significantly through the introduction of semantically composite examples during pretraining. Inspired by CutMix in vision categorization, we create semantically composite image-caption pairs by merging elements from two distinct instances in the dataset via a novel procedure. Our method fuses the captions and blends 50% of each image to form a new composite sample. This simple technique (termed CLIP-C for CLIP Compositions), devoid of any additional computational overhead or increase in model parameters, significantly improves zero-shot image classification and cross-modal retrieval. The benefits of CLIP-C are particularly pronounced in settings with relatively limited pretraining data.

📄 PDF Abstract BibTeX arXiv:2407.01408

Code (0)

등록된 구현이 없습니다.

Tasks

Classification Contrastive Learning Cross-Modal Retrieval image-classification Image ClassificationLinear evaluationRetrievalzero-shot-classificationZero-Shot Image Classification Zero-Shot Learning

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

CutMix CutMix is an image data augmentation strategy. Instead of simply removing pixels as in Cutout, we replace the removed regions with…

Anticipating Future Object Compositions without Forgetting

2024-07-15 · Youssef Zahran, Gertjan Burghouts, Yke Bauke Eisma

Despite the significant advancements in computer vision models, their ability to generalize to novel object-attribute compositions remains limited. Existing methods for Compositional Zero-Shot Learning (CZSL) mainly focu…

Attribute Compositional Zero-Shot Learning image-classification Image Classification+4

DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations

2026-08-20 · Hangyu Tian, Zhenqi He, Yanghao Wang, Long Chen arxiv

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive perfor…

Compositional Zero-Shot Learning

Multi-Sourced Compositional Generalization in Visual Question Answering

2025-05-29 · Chuanhao Li, Wenbo Ye, Zhen Li, Yuwei Wu 외

Compositional generalization is the ability of generalizing novel compositions from seen primitives, and has received much attention in vision-and-language (V\&L) recently. Due to the multi-modal nature of V\&L tasks, th…

Question Answering Visual Question Answering Visual Question Answering (VQA)

CAMS: Towards Compositional Zero-Shot Learning via Gated Cross-Attention and Multi-Space Disentanglement

2025-11-20 · Pan Yang, Cheng Deng, Jing Yang, Han Zhao 외 arxiv

Compositional zero-shot learning (CZSL) aims to learn the concepts of attributes and objects in seen compositions and to recognize their unseen compositions. Most Contrastive Language-Image Pre-training (CLIP)-based CZSL…