Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
Pre-trained contextual vision-and-language (V&L) models have achieved impressive performance on various benchmarks. However, existing models require a large amount of parallel image-caption data for pre-training. Such data are costly to collect and require cumbersome curation. Inspired by unsupervised machine translation, we investigate if a strong V&L representation model can be learned through unsupervised pre-training without image-caption corpora. In particular, we propose to conduct ``mask-and-predict'' pre-training on text-only and image-only corpora and introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. We find that such a simple approach achieves performance close to a model pre-trained with aligned data, on four English V&L benchmarks. Our work challenges the widely held notion that aligned data is necessary for V&L pre-training, while significantly reducing the amount of supervision needed for V&L models.
Code (1)
Tasks
Machine TranslationObject RecognitionTranslationUnsupervised Machine TranslationUnsupervised Pre-trainingSimilar Papers 제목 키워드 기반
Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignment
Vision-and-Language (V+L) pre-training models have achieved tremendous success in recent years on various multi-modal benchmarks. However, the majority of existing models require pre-training on a large set of parallel i…
RetrievalSentenceTAGVisual Entailment+1Parallel sentences mining with transfer learning in an unsupervised setting
The quality and quantity of parallel sentences are known as very important training data for constructing neural machine translation (NMT) systems. However, these resources are not available for many low-resource languag…
Machine TranslationNMTTransfer LearningTranslationIt's a (Blind) Match! Towards Vision-Language Correspondence without Parallel Data
The platonic representation hypothesis suggests that vision and language embeddings become more homogeneous as model and dataset sizes increase. In particular, pairwise distances within each modality become more similar.…
text annotationUnsupervised Parallel Sentence Extraction from Comparable Corpora
Mining parallel sentences from comparable corpora is of great interest for many downstream tasks. In the BUCC 2017 shared task, systems performed well by training on gold standard parallel sentences. However, we often wa…
SentenceWord EmbeddingsWord Translation Without Parallel Data
State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with char…
Cross-Lingual Word EmbeddingsMachine TranslationTranslationUnsupervised Machine Translation+3