Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining
Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. In this work, we take a step further and explore how we can tap into supervision from small-scale visual relation data. In particular, we propose two pretraining approaches to contextualise visual entities in a multimodal setup. With verbalised scene graphs, we transform visual relation triplets into structured captions, and treat them as additional image descriptions. With masked relation prediction, we further encourage relating entities from image regions with visually masked contexts. When applied to strong baselines pretrained on large amounts of Web data, zero-shot evaluations on both coarse-grained and fine-grained tasks show the efficacy of our methods in learning multimodal representations from weakly-supervised relations data.
Code (1)
Tasks
object-detectionObject DetectionRelationRelation PredictionWeakly-supervised LearningSimilar Papers 제목 키워드 기반
Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations
Large-scale pretraining of visual representations has led to state-of-the-art performance on a range of benchmark computer vision tasks, yet the benefits of these techniques at extreme scale in complex production systems…
Image ClassificationMulti-Task LearningRepresentation LearningRetrievalExploring the Limits of Weakly Supervised Pretraining
State-of-the-art visual perception models for a wide range of tasks rely on supervised pretraining. ImageNet classification is the de facto pretraining task for these models. Yet, ImageNet is now nearly ten years old and…
General Classificationimage-classificationImage Classificationobject-detection+2Multimodal Semi-Supervised Learning for Text Recognition
Until recently, the number of public real-world text images was insufficient for training scene text recognizers. Therefore, most modern training methods rely on synthetic data and operate in a fully supervised manner. N…
Language ModellingRepresentation LearningScene Text RecognitionSelf-Supervised LearningMAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
Phrase localization is a task that studies the mapping from textual phrases to regions of an image. Given difficulties in annotating phrase-to-object datasets at scale, we develop a Multimodal Alignment Framework (MAF) t…
Phrase GroundingRIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER
Multimodal named entity recognition (MNER) for tweets has received increasing attention recently. Most of the multimodal methods used attention mechanisms to capture the text-related visual information. However, unrelate…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER+2