Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP

2023-10-02 · Zixiang Chen, Yihe Deng, Yuanzhi Li, Quanquan Gu

Multi-modal learning has become increasingly popular due to its ability to leverage information from different data sources (e.g., text and images) to improve the model performance. Recently, CLIP has emerged as an effective approach that employs vision-language contrastive pretraining to learn joint image and text representations and exhibits remarkable performance in zero-shot learning and text-guided natural image generation. Despite the huge practical success of CLIP, its theoretical understanding remains elusive. In this paper, we formally study transferrable representation learning underlying CLIP and demonstrate how features from different modalities get aligned. We also analyze its zero-shot transfer performance on the downstream tasks. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-the-art methods on benchmark datasets.

📄 PDF Abstract BibTeX arXiv:2310.00927

Code (0)

등록된 구현이 없습니다.

Tasks

Image Generation Representation Learning Zero-Shot Learning

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition

2026-06-19 · Di Yang, Mahmoud Ali, Quan Kong, Gianpiero Francesca 외 arxiv

Vision-language models such as CLIP have recently achieved strong performance on a wide range of visual understanding tasks. However, most existing models rely primarily on appearance-level supervision from images or vid…

Action Classification Contrastive Learning Action Understanding Action Recognition

Task-Independent Knowledge Makes for Transferable Representations for Generalized Zero-Shot Learning

2021-04-05 · Chaoqun Wang, Xuejin Chen, Shaobo Min, Xiaoyan Sun 외

Generalized Zero-Shot Learning (GZSL) targets recognizing new categories by learning transferable image representations. Existing methods find that, by aligning image representations with corresponding semantic labels, t…

Contrastive Learning Generalized Zero-Shot Learning Zero-Shot Learning

Zero-shot Cross-lingual Dialogue Systems with Transferable Latent Variables

2019-11-11 · IJCNLP 2019 11 · Zihan Liu, Jamin Shin, Yan Xu, Genta Indra Winata 외

Despite the surging demands for multilingual task-oriented dialog systems (e.g., Alexa, Google Home), there has been less research done in multilingual or cross-lingual scenarios. Hence, we propose a zero-shot adaptation…

Intent Detection Natural Language Understanding slot-filling Slot Filling

Learning Transferable Sensor Models via Language-Informed Pretraining

2026-03-12 · Yuliang Chen, Arvind Pillai, Yu Yvonne Wu, Tess Z. Griffin 외 arxiv

Modern sensing systems generate large volumes of unlabeled multivariate time-series data. This abundance of unlabeled data makes self-supervised learning (SSL) a natural approach for learning transferable representations…

Self-Supervised Learning Question Answering

Zero-shot stance detection based on cross-domain feature enhancement by contrastive learning

2022-10-07 · Xuechen Zhao, Jiaying Zou, Zhong Zhang, Feng Xie 외

Zero-shot stance detection is challenging because it requires detecting the stance of previously unseen targets in the inference phase. The ability to learn transferable target-invariant features is critical for zero-sho…

Contrastive Learning Stance Detection Zero-Shot Stance Detection