Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

2021-04-18 · Ruizhe Cheng, Bichen Wu, Peizhao Zhang, Peter Vajda, Joseph E. Gonzalez

Traditional computer vision models are trained to predict a fixed set of predefined categories. Recently, natural language has been shown to be a broader and richer source of supervision that provides finer descriptions to visual concepts than supervised "gold" labels. Previous works, such as CLIP, use a simple pretraining task of predicting the pairings between images and text captions. CLIP, however, is data hungry and requires more than 400M image text pairs for training. We propose a data-efficient contrastive distillation method that uses soft labels to learn from noisy image-text pairs. Our model transfers knowledge from pretrained image and sentence encoders and achieves strong performance with only 3M image text pairs, 133x smaller than CLIP. Our method exceeds the previous SoTA of general zero-shot learning on ImageNet 21k+1k by 73% relatively with a ResNet50 image encoder and DeCLUTR text encoder. We also beat CLIP by 10.5% relatively on zero-shot evaluation on Google Open Images (19,958 classes).

📄 PDF Abstract BibTeX arXiv:2104.08945

Code (0)

등록된 구현이 없습니다.

Tasks

SentenceZero-Shot Learning

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

DeCLUTR DeCLUTR is an approach for learning universal sentence embeddings that utilizes a self-supervised objective that does not require labelled training data. The objective learns…

Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models

2021-09-10 · EMNLP 2021 11 · Tassilo Klein, Moin Nabi

Can we get existing language models and refine them for zero-shot commonsense reasoning? This paper presents an initial study exploring the feasibility of zero-shot commonsense reasoning for the Winograd Schema Challenge…

Language Modeling Language Modelling Self-Supervised Learning

Go-tuning: Improving Zero-shot Learning Abilities of Smaller Language Models

2022-12-20 · Jingjing Xu, Qingxiu Dong, Hongyi Liu, Lei LI

With increasing scale, large language models demonstrate both quantitative improvement and new qualitative capabilities, especially as zero-shot learners, like GPT-3. However, these results rely heavily on delicate promp…

Language Modeling Language Modelling Masked Language Modeling Self-Supervised Learning+1

Self-Learning for Zero Shot Neural Machine Translation

2021-03-10 · Surafel M. Lakew, Matteo Negri, Marco Turchi

Neural Machine Translation (NMT) approaches employing monolingual data are showing steady improvements in resource rich conditions. However, evaluations using real-world low-resource languages still result in unsatisfact…

Machine Translation NMT Self-Learning Translation

Zero-Shot Neural Machine Translation with Self-Learning Cycle

2021-08-01 · MTSummit 2021 8 · Surafel M. Lakew, Matteo Negri, Marco Turchi

Neural Machine Translation (NMT) approaches employing monolingual data are showing steady improvements in resource-rich conditions. However, evaluations using real-world lowresource languages still result in unsatisfacto…

Machine Translation NMT Self-Learning Translation

Zero-Shot Text Classification via Self-Supervised Tuning

2023-05-19 · Chaoqun Liu, Wenxuan Zhang, Guizhen Chen, Xiaobao Wu 외

Existing solutions to zero-shot text classification either conduct prompting with pre-trained language models, which is sensitive to the choices of templates, or rely on large-scale annotated data of relevant tasks for m…

Classification Self-Supervised Learning Sentence Sentiment Analysis+4