Dense Contrastive Visual-Linguistic Pretraining
Inspired by the success of BERT, several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. In particular, LXMERT and UNITER adopt visual region feature regression and label classification as pretext tasks. However, they tend to suffer from the problems of noisy labels and sparse semantic annotations, based on the visual features having been pretrained on a crowdsourced dataset with limited and inconsistent semantic labeling. To overcome these issues, we propose unbiased Dense Contrastive Visual-Linguistic Pretraining (DCVLP), which replaces the region regression and classification with cross-modality region contrastive learning that requires no annotations. Two data augmentation strategies (Mask Perturbation and Intra-/Inter-Adversarial Perturbation) are developed to improve the quality of negative samples used in contrastive learning. Overall, DCVLP allows cross-modality dense region contrastive learning in a self-supervised setting independent of any object annotations. We compare our method against prior visual-linguistic pretraining frameworks to validate the superiority of dense contrastive learning on multimodal representation learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningData AugmentationregressionRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Contrastive Visual-Linguistic Pretraining
Several multi-modality representation learning approaches such as LXMERT and ViLBERT have been proposed recently. Such approaches can achieve superior performance due to the high-level semantic information captured durin…
Contrastive LearningregressionRepresentation LearningVisual Question Answering (VQA)LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
Vision-language pretraining remains dominated by contrastive objectives, whereas vision-only self-supervised learning has largely adopted non-contrastive methods. At the same time, the role of vision-language encoders ha…
Self-Supervised LearningSemantic SegmentationUnsupervised Learning of Dense Visual Representations
Contrastive self-supervised learning has emerged as a promising approach to unsupervised visual representation learning. In general, these methods learn global (image-level) representations that are invariant to differen…
Contrastive LearningData AugmentationRepresentation LearningSelf-Supervised LearningFLAVARS: A Multimodal Foundational Language and Vision Alignment Model for Remote Sensing
Remote sensing imagery is dense with objects and contextual visual information. There is a recent trend to combine paired satellite images and text captions for pretraining performant encoders for downstream tasks. Howev…
ClassificationContrastive LearningSemantic Segmentationzero-shot-classification+1Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning
We propose to solve the natural language inference problem without any supervision from the inference labels via task-agnostic multimodal pretraining. Although recent studies of multimodal self-supervised learning also r…
Contrastive LearningNatural Language InferenceSelf-Supervised LearningSTS+1