A Confidence-Based Interface for Neuro-Symbolic Visual Question Answering
We present a neuro-symbolic visual question answering (VQA) approach for the CLEVR dataset that is based on the combination of deep neural networks and answer-set programming (ASP), a logic-based paradigm for declarative problem solving. We provide a translation mechanism for the questions included in CLEVR to ASP programs. By exploiting choice rules, we consider deterministic and non-deterministic scene encodings. In addition, we introduce a confidence-based interface between the ASP module and the neural network which allows us to restrict the non-determinism to objects classified by the network with high confidence. Our experiments show that the non-deterministic scene encoding achieves good results even if the neural networks are trained rather poorly in comparison with the deterministic approach. This is important for building robust VQA systems if network predictions are less-than perfect.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringTranslationVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
A Neuro-Symbolic ASP Pipeline for Visual Question Answering
We present a neuro-symbolic visual question answering (VQA) pipeline for CLEVR, which is a well-known dataset that consists of pictures showing scenes with objects and questions related to them. Our pipeline covers (i) t…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Answer Set Programming Energised! End-to-End Neurosymbolic Reasoning and Learning with ASP and Energy Based Models
We present a general neurosymbolic reasoning and learning methodology based on a modular integration of answer set programming with an energy based model substrate. Key contributions are: (1) supporting joint optimisatio…
Multi-Object TrackingNeuro-Symbolic Visual Dialog
We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connect…
Question AnsweringThe Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply lookin…
Image-text RetrievalObjectQuestion AnsweringRetrieval+4An Interactive Web-Interface for Visualizing the Inner Workings of the Question Answering LSTM
We present a visualisation tool which aims to illuminate the inner workings of an LSTM model for question answering. It plots heatmaps of neurons{'} firings and allows a user to check the dependency between neurons and m…
Feature EngineeringMachine TranslationQuestion Answering