Compact Trilinear Interaction for Visual Question Answering
In Visual Question Answering (VQA), answers have a great correlation with question meaning and visual contents. Thus, to selectively utilize image, question and answer information, we propose a novel trilinear interaction model which simultaneously learns high level associations between these three inputs. In addition, to overcome the interaction complexity, we introduce a multimodal tensor-based PARALIND decomposition which efficiently parameterizes trilinear interaction between the three inputs. Moreover, knowledge distillation is first time applied in Free-form Opened-ended VQA. It is not only for reducing the computational cost and required memory but also for transferring knowledge from trilinear interaction model to bilinear interaction model. The extensive experiments on benchmarking datasets TDIUC, VQA-2.0, and Visual7W show that the proposed compact trilinear interaction model achieves state-of-the-art results when using a single model on all three datasets.
Code (1)
Tasks
BenchmarkingKnowledge DistillationQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MIRTT: Learning Multimodal Interaction Representations from Trilinear Transformers for Visual Question Answering
In Visual Question Answering (VQA), existing bilinear methods focus on the interaction between images and questions. As a result, the answers are either spliced into the questions or utilized as labels only for classific…
multimodal interactionMultiple-choiceQuestion AnsweringVisual Question Answering+1Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require c…
Phrase GroundingVisual GroundingVisual Question AnsweringVisual Question Answering (VQA)Kernel Pooling for Convolutional Neural Networks
Convolutional Neural Networks (CNNs) with Bilinear Pooling, initially in their full form and later using compact representations, have yielded impressive performance gains on a wide range of visual tasks, including fine-…
Face RecognitionFine-Grained Visual CategorizationQuestion AnsweringVisual Question Answering+1Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering
This paper presents a novel method, termed Bridge to Answer, to infer correct answers for questions about a given video by leveraging adequate graph interactions of heterogeneous crossmodal graphs. To realize this, we le…
Question AnsweringVideo Question AnsweringCompact Tensor Pooling for Visual Question Answering
Performing high level cognitive tasks requires the integration of feature maps with drastically different structure. In Visual Question Answering (VQA) image descriptors have spatial structures, while lexical inputs inhe…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)