VLMAE: Vision-Language Masked Autoencoder
Image and language modeling is of crucial importance for vision-language pre-training (VLP), which aims to learn multi-modal representations from large-scale paired image-text data. However, we observe that most existing VLP methods focus on modeling the interactions between image and text features while neglecting the information disparity between image and text, thus suffering from focal bias. To address this problem, we propose a vision-language masked autoencoder framework (VLMAE). VLMAE employs visual generative learning, facilitating the model to acquire fine-grained and unbiased features. Unlike the previous works, VLMAE pays attention to almost all critical patches in an image, providing more comprehensive understanding. Extensive experiments demonstrate that VLMAE achieves better performance in various vision-language downstream tasks, including visual question answering, image-text retrieval and visual grounding, even with up to 20% pre-training speedup.
Code (0)
등록된 구현이 없습니다.
Tasks
Image-text RetrievalLanguage ModelingLanguage ModellingQuestion AnsweringRetrievalText RetrievalVisual GroundingVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond
Masked autoencoders are scalable vision learners, as the title of MAE \cite{he2022masked}, which suggests that self-supervised learning (SSL) in vision might undertake a similar trajectory as in NLP. Specifically, genera…
Contrastive LearningDenoisingSelf-Supervised LearningQuantum Masked Autoencoders for Vision Learning
Classical autoencoders are widely used to learn features of input data. To improve the feature learning, classical masked autoencoders extend classical autoencoders to learn the features of the original input sample in t…
RARE: Robust Masked Graph Autoencoder
Masked graph autoencoder (MGAE) has emerged as a promising self-supervised graph pre-training (SGP) paradigm due to its simplicity and effectiveness. However, existing efforts perform the mask-then-reconstruct operation …
Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training
Medical vision-and-language pre-training provides a feasible solution to extract effective vision-and-language representations from medical images and texts. However, few studies have been dedicated to this field to faci…
Self-Supervised LearningMILAN: Masked Image Pretraining on Language Assisted Representation
Self-attention based transformer models have been dominating many computer vision tasks in the past few years. Their superb model qualities heavily depend on the excessively large labeled image datasets. In order to redu…
DecoderSemantic Segmentation