Grounded Image Text Matching with Mismatched Relation Reasoning
This paper introduces Grounded Image Text Matching with Mismatched Relation (GITM-MR), a novel visual-linguistic joint task that evaluates the relation understanding capabilities of transformer-based pre-trained models. GITM-MR requires a model to first determine if an expression describes an image, then localize referred objects or ground the mismatched parts of the text. We provide a benchmark for evaluating pre-trained models on this task, with a focus on the challenging settings of limited data and out-of-distribution sentence lengths. Our evaluation demonstrates that pre-trained models lack data efficiency and length generalization ability. To address this, we propose the Relation-sensitive Correspondence Reasoning Network (RCRN), which incorporates relation-aware reasoning via bi-directional message propagation guided by language structure. RCRN can be interpreted as a modular program and delivers strong performance in both length generalization and data efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
Image-text matchingRelationSentenceText MatchingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Negative-Aware Attention Framework for Image-Text Matching
Image-text matching, as a fundamental task, bridges the gap between vision and language. The key of this task is to accurately measure similarity between these two modalities. Prior work measuring this similarity mai…
Image-text matchingText Matchingtext similarityREPAIR: Rank Correlation and Noisy Pair Half-replacing with Memory for Noisy Correspondence
The presence of noise in acquired data invariably leads to performance degradation in cross-modal matching. Unfortunately, obtaining precise annotations in the multimodal field is expensive, which has prompted some metho…
Cross-modal retrieval with noisy correspondenceGrounding Language Representation with Visual Object Information via Cross Modal Pretraining
Previous studies of visual grounded language learning use a convolutional neural network (CNN) to extract features from the whole image for grounding with the sentence description. However, this approach has two main dra…
Grounded language learningObjectSentenceConstructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval
Existing research for image text retrieval mainly relies on sentence-level supervision to distinguish matched and mismatched sentences for a query image. However, semantic mismatch between an image and sentences usually …
FormImage-text RetrievalRepresentation LearningRetrieval+2Constructing Phrase-level Semantic Labels to Form Multi-GrainedSupervision for Image-Text Retrieval
Existing research for image text retrieval mainly relies on sentence-level supervision to distinguish matched and mismatched sentences for a query image. However, semantic mismatch between an image and sentences usually …
FormImage-text RetrievalRepresentation LearningRetrieval+2