Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval
Image-text retrieval of natural scenes has been a popular research topic. Since image and text are heterogeneous cross-modal data, one of the key challenges is how to learn comprehensive yet unified representations to express the multi-modal data. A natural scene image mainly involves two kinds of visual concepts, objects and their relationships, which are equally essential to image-text retrieval. Therefore, a good representation should account for both of them. In the light of recent success of scene graph in many CV and NLP tasks for describing complex natural scenes, we propose to represent image and text with two kinds of scene graphs: visual scene graph (VSG) and textual scene graph (TSG), each of which is exploited to jointly characterize objects and relationships in the corresponding modality. The image-text retrieval task is then naturally formulated as cross-modal scene graph matching. Specifically, we design two particular scene graph encoders in our model for VSG and TSG, which can refine the representation of each node on the graph by aggregating neighborhood information. As a result, both object-level and relationship-level cross-modal features can be obtained, which favorably enables us to evaluate the similarity of image and text in the two levels in a more plausible way. We achieve state-of-the-art results on Flickr30k and MSCOCO, which verifies the advantages of our graph matching based approach for image-text retrieval.
Code (1)
Tasks
Graph MatchingImage-text RetrievalRetrievalText RetrievalSimilar Papers 제목 키워드 기반
SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs
We introduce a novel problem, i.e., the localization of an input image within a multi-modal reference map represented by a database of 3D scene graphs. These graphs comprise multiple modalities, including object-level po…
Visual LocalizationHire: Hybrid-modal Interaction with Multiple Relational Enhancements for Image-Text Matching
Image-text matching (ITM) is a fundamental problem in computer vision. The key issue lies in jointly learning the visual and textual representation to estimate their similarity accurately. Most existing methods focus on …
cross-modal alignmentImage-text matchingRelationship DetectionText MatchingHyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation
Multimodal LLMs have advanced vision-language tasks but still struggle with understanding video scenes. To bridge this gap, Video Scene Graph Generation (VidSGG) has emerged to capture multi-object relationships across v…
Graph GenerationQuestion AnsweringScene Graph GenerationVideo Captioning+2MOSA: Motion-Guided Semantic Alignment for Dynamic Scene Graph Generation
Dynamic Scene Graph Generation (DSGG) aims to structurally model objects and their dynamic interactions in video sequences for high-level semantic understanding. However, existing methods struggle with fine-grained relat…
Scene Graph GenerationBilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering
Answering semantically-complicated questions according to an image is challenging in Visual Question Answering (VQA) task. Although the image can be well represented by deep learning, the question is always simply embedd…
Graph MatchingQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)