Context-Aware Attention Network for Image-Text Retrieval
As a typical cross-modal problem, image-text bi-directional retrieval relies heavily on the joint embedding learning and similarity measure for each image-text pair. It remains challenging because prior works seldom explore semantic correspondences between modalities and semantic correlations in a single modality at the same time. In this work, we propose a unified Context-Aware Attention Network (CAAN), which selectively focuses on critical local fragments (regions and words) by aggregating the global context. Specifically, it simultaneously utilizes global inter-modal alignments and intra-modal correlations to discover latent semantic relations. Considering the interactions between images and sentences in the retrieval process, intra-modal correlations are derived from the second-order attention of region-word alignments instead of intuitively comparing the distance between original features. Our method achieves fairly competitive results on two generic image-text retrieval datasets Flickr30K and MS-COCO.
Code (0)
등록된 구현이 없습니다.
Tasks
Image-text RetrievalRetrievalText RetrievalSimilar Papers 제목 키워드 기반
AttentionRetriever: Attention Layers are Secretly Long Document Retrievers
Retrieval augmented generation (RAG) has been widely adopted to help Large Language Models (LLMs) to process tasks involving long documents. However, existing retrieval models are not designed for long document retrieval…
Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval
Composed Image Retrieval (CIR) has demonstrated significant potential by enabling flexible multimodal queries that combine a reference image and modification text. However, CIR inherently prioritizes semantic matching, s…
Image RetrievalStructured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval
The current state-of-the-art image-sentence retrieval methods implicitly align the visual-textual fragments, like regions in images and words in sentences, and adopt attention modules to highlight the relevance of cross-…
cross-modal alignmentRetrievalSemantic correspondenceSentence+1Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection
The goal of video moment retrieval and highlight detection is to identify specific segments and highlights based on a given text query. With the rapid growth of video content and the overlap between these tasks, recent w…
Contrastive LearningHighlight DetectionMoment RetrievalRetrievalBoolean-aware Attention for Dense Retrieval
We present Boolean-aware attention, a novel attention mechanism that dynamically adjusts token focus based on Boolean operators (e.g., and, or, not). Our model employs specialized Boolean experts, each tailored to amplif…
Retrieval