Multi-label Cluster Discrimination for Visual Representation Learning
Contrastive Language Image Pre-training (CLIP) has recently demonstrated success across various tasks due to superior feature representation empowered by image-text contrastive learning. However, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data. To handle this limitation, cluster discrimination has been proposed through iterative cluster assignment and classification. Nevertheless, most cluster discrimination approaches only define a single pseudo-label for each image, neglecting multi-label signals in the image. In this paper, we propose a novel Multi-Label Cluster Discrimination method named MLCD to enhance representation learning. In the clustering step, we first cluster the large-scale LAION-400M dataset into one million centers based on off-the-shelf embedding features. Considering that natural images frequently contain multiple visual objects or attributes, we select the multiple closest centers as auxiliary class labels. In the discrimination step, we design a novel multi-label classification loss, which elegantly separates losses from positive classes and negative classes, and alleviates ambiguity on decision boundary. We validate the proposed multi-label cluster discrimination method with experiments on different scales of models and pre-training datasets. Experimental results show that our method achieves state-of-the-art performance on multiple downstream tasks including linear probe, zero-shot classification, and image-text retrieval. Code and models have been released at https://github.com/deepglint/unicom .
Code (1)
Tasks
Contrastive LearningImage-text RetrievalMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONPseudo LabelReferring Expression SegmentationRepresentation LearningSelf-Supervised Image ClassificationText RetrievalVisual Question Answering (VQA)zero-shot-classificationZero-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unsupervised Visual Representation Learning by Online Constrained K-Means
Cluster discrimination is an effective pretext task for unsupervised representation learning, which often consists of two phases: clustering and discrimination. Clustering is to assign each instance a pseudo label that w…
ClusteringContrastive LearningImage ClusteringMetric Learning+5Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data
This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable repres…
Contrastive LearningRepresentation LearningA Flexible Framework for Discovering Novel Categories with Contrastive Learning
This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable repres…
Contrastive LearningRepresentation LearningRegion-based Cluster Discrimination for Visual Representation Learning
Learning visual representations is foundational for a broad spectrum of downstream tasks. Although recent vision-language contrastive models, such as CLIP and SigLIP, have achieved impressive zero-shot performance via la…
Representation LearningTask-Independent Knowledge Makes for Transferable Representations for Generalized Zero-Shot Learning
Generalized Zero-Shot Learning (GZSL) targets recognizing new categories by learning transferable image representations. Existing methods find that, by aligning image representations with corresponding semantic labels, t…
Contrastive LearningGeneralized Zero-Shot LearningZero-Shot Learning