Reciprocal Attention Fusion for Visual Question Answering
Existing attention mechanisms either attend to local image grid or object level features for Visual Question Answering (VQA). Motivated by the observation that questions can relate to both object instances and their parts, we propose a novel attention mechanism that jointly considers reciprocal relationships between the two levels of visual details. The bottom-up attention thus generated is further coalesced with the top-down information to only focus on the scene elements that are most relevant to a given question. Our design hierarchically fuses multi-modal information i.e., language, object- and gird-level features, through an efficient tensor decomposition scheme. The proposed model improves the state-of-the-art single model performances from 67.9% to 68.2% on VQAv1 and from 65.7% to 67.4% on VQAv2, demonstrating a significant boost.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectQuestion AnsweringTensor DecompositionVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation
Neural models that independently project questions and answers into a shared embedding space allow for efficient continuous space retrieval from large corpora. Independently computing embeddings for questions and answers…
Data AugmentationQuestion AnsweringRetrievalEnhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusion
Visual Question Answering (VQA) is a challenging task that requires systems to provide accurate answers to questions based on image content. Current VQA models struggle with complex questions due to limitations in captur…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Microsoft AI Challenge India 2018: Learning to Rank Passages for Web Question Answering with Deep Attention Networks
This paper describes our system for The Microsoft AI Challenge India 2018: Ranking Passages for Web Question Answering. The system uses the biLSTM network with co-attention mechanism between query and passage representat…
Deep AttentionLearning-To-RankQuestion AnsweringDynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
Learning effective fusion of multi-modality features is at the heart of visual question answering. We propose a novel method of dynamically fusing multi-modal features with intra- and inter-modality information flow, whi…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering
Learning effective fusion of multi-modality features is at the heart of visual question answering. We propose a novel method of dynamically fuse multi-modal features with intra- and inter-modality information flow, which…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)