Compositional Memory for Visual Question Answering
Visual Question Answering (VQA) emerges as one of the most fascinating topics in computer vision recently. Many state of the art methods naively use holistic visual features with language features into a Long Short-Term Memory (LSTM) module, neglecting the sophisticated interaction between them. This coarse modeling also blocks the possibilities of exploring finer-grained local features that contribute to the question answering dynamically over time. This paper addresses this fundamental problem by directly modeling the temporal dynamics between language and all possible local image patches. When traversing the question words sequentially, our end-to-end approach explicitly fuses the features associated to the words and the ones available at multiple local patches in an attention mechanism, and further combines the fused information to generate dynamic messages, which we call episode. We then feed the episodes to a standard question answering module together with the contextual visual information and linguistic information. Motivated by recent practices in deep learning, we use auxiliary loss functions during training to improve the performance. Our experiments on two latest public datasets suggest that our method has a superior performance. Notably, on the DARQUAR dataset we advanced the state of the art by 6$\%$, and we also evaluated our approach on the most recent MSCOCO-VQA dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Compositional Attention Networks for Interpretability in Natural Language Question Answering
MAC Net is a compositional attention network designed for Visual Question Answering. We propose a modified MAC net architecture for Natural Language Question Answering. Question Answering typically requires Language Unde…
Logical ReasoningQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)MacVQA: Adaptive Memory Allocation and Global Noise Filtering for Continual Visual Question Answering
Visual Question Answering (VQA) requires models to reason over multimodal information, combining visual and textual data. With the development of continual learning, significant progress has been made in retaining knowle…
Visual Question AnsweringContinual LearningC-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
Visual Question Answering (VQA) has received a lot of attention over the past couple of years. A number of deep learning models have been proposed for this task. However, it has been shown that these models are heavily d…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Maintaining Reasoning Consistency in Compositional Visual Question Answering
A compositional question refers to a question that contains multiple visual concepts (e.g., objects, attributes, and relationships) and requires compositional reasoning to answer. Existing VQA models can answer a com…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Interpretable Neural Computation for Real-World Compositional Visual Question Answering
There are two main lines of research on visual question answering (VQA): compositional model with explicit multi-hop reasoning, and monolithic network with implicit reasoning in the latent feature space. The former excel…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)