Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.
Code (1)
Tasks
Cross-Modal RetrievalImage CaptioningImage RetrievalObjectobject-detectionObject DetectionOpen Vocabulary Attribute DetectionReferring Expression SegmentationVisual GroundingVisual Question Answering (VQA)Visual ReasoningSimilar Papers 제목 키워드 기반
Curriculum Learning for Data-Efficient Vision-Language Alignment
Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation mo…
Contrastive LearningImage RetrievalObjectRetrieval+1X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
Vision language pre-training aims to learn alignments between vision and language from a large amount of data. Most existing methods only learn image-text alignments. Some others utilize pre-trained object detectors to l…
AllCross-Modal RetrievalImage CaptioningText to Video Retrieval+6Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) excel in vision--language tasks by pre-training solely on coarse-grained concept annotations (e.g., image captions). We hypothesize that integrating fine-grained concept annotatio…
Image Captioning3D Vision and Language Pretraining with Large-Scale Synthetic Data
3D Vision-Language Pre-training (3D-VLP) aims to provide a pre-train model which can bridge 3D scenes with natural language, which is an important technique for embodied intelligence. However, current 3D-VLP datasets are…
Dense CaptioningDiversityDomain AdaptationQuestion Answering+1SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
Vision-language pre-training (VLP) on large-scale image-text pairs has recently witnessed rapid progress for learning cross-modal representations. Existing pre-training methods either directly concatenate image represent…