BERT-VQA: Visual Question Answering on Plots
Visual question answering has been an exciting challenge in the field of natural language understanding, as it requires deep learning models to exchange information from both vision and language domains. In this project, we aim to tackle a subtask of this problem, namely visual question answering on plots. To achieve this, we developed BERT-VQA, a VisualBERT-based model architecture with a pretrained ResNet 101 image encoder, along with a potential addition of joint fusion. We trained and evaluated this model against a baseline that consisted of a LSTM, a CNN, and a shallow classifier. The final outcome disproved our core hypothesis that the cross-modality module in VisualBERT is essential in aligning plot components with question phrases. Therefore, our work provided valuable insights into the difficulty of the plot question answering challenge as well as the appropriateness of different model architectures in solving this problem.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingVisual Question AnsweringSimilar Papers 제목 키워드 기반
An Interactive Web-Interface for Visualizing the Inner Workings of the Question Answering LSTM
We present a visualisation tool which aims to illuminate the inner workings of an LSTM model for question answering. It plots heatmaps of neurons{'} firings and allows a user to check the dependency between neurons and m…
Feature EngineeringMachine TranslationQuestion AnsweringDCQA: Document-Level Chart Question Answering towards Complex Reasoning and Common-Sense Understanding
Visually-situated languages such as charts and plots are omnipresent in real-world documents. These graphical depictions are human-readable and are often analyzed in visually-rich documents to address a variety of questi…
Answer GenerationChart Question AnsweringCommon Sense ReasoningDocument Layout Analysis+3Hierarchical Visual Agent: Managing Contexts in Joint Image-Text Space for Advanced Chart Reasoning
Advanced chart question answering requires both precise perception of small visual elements and multi-step reasoning across several subplots. While existing MLLMs are strong at understanding single plots, they often stru…
Chart Question AnsweringFigureNet: A Deep Learning model for Question-Answering on Scientific Plots
Deep Learning has managed to push boundaries in a wide variety of tasks. One area of interest is to tackle problems in reasoning and understanding, with an aim to emulate human intelligence. In this work, we describe a d…
Deep LearningQuestion AnsweringMIMOQA: Multimodal Input Multimodal Output Question Answering
Multimodal research has picked up significantly in the space of question answering with the task being extended to visual question answering, charts question answering as well as multimodal input question answering. Howe…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)