Learning Differentiable Logic Programs for Abstract Visual Reasoning
Visual reasoning is essential for building intelligent agents that understand the world and perform problem-solving beyond perception. Differentiable forward reasoning has been developed to integrate reasoning with gradient-based machine learning paradigms. However, due to the memory intensity, most existing approaches do not bring the best of the expressivity of first-order logic, excluding a crucial ability to solve abstract visual reasoning, where agents need to perform reasoning by using analogies on abstract concepts in different scenarios. To overcome this problem, we propose NEUro-symbolic Message-pAssiNg reasoNer (NEUMANN), which is a graph-based differentiable forward reasoner, passing messages in a memory-efficient manner and handling structured programs with functors. Moreover, we propose a computationally-efficient structure learning algorithm to perform explanatory program induction on complex visual scenes. To evaluate, in addition to conventional visual reasoning tasks, we propose a new task, visual reasoning behind-the-scenes, where agents need to learn abstract programs and then answer queries by imagining scenes that are not observed. We empirically demonstrate that NEUMANN solves visual reasoning tasks efficiently, outperforming neural, symbolic, and neuro-symbolic baselines.
Code (1)
Tasks
Program inductionVisual ReasoningSimilar Papers 제목 키워드 기반
Differentiable End-to-End Program Executor for Sample and Computationally Efficient VQA
We present a differentiable end-to-end program executor (DePe), which addresses Visual Question Answering (VQA) in a sample and computationally efficient manner. DePe parses the question into probabilistic programs and s…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Learning Rules with Stratified Negation in Differentiable ILP.
Differentiable methods to learn rules (logic programs) have the potential to integrate the interpretability, transferability and low data requirements of inductive logic programming with the noise tolerance of non-symbol…
Inductive logic programmingNegationDifferentiable Inductive Logic Programming for Structured Examples
The differentiable implementation of logic yields a seamless combination of symbolic reasoning and deep neural networks. Recent research, which has developed a differentiable framework to learn logic programs from exampl…
Inductive logic programmingLearning logic programs by discovering higher-order abstractions
We introduce the higher-order refactoring problem, where the goal is to compress a logic program by discovering higher-order abstractions, such as map, filter, and fold. We implement our approach in Stevie, which formula…
Inductive logic programmingProgram SynthesisVisual ReasoningWhat's Left? Concept Grounding with Logic-Enhanced Foundation Models
Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning-using large language models (LLMs) to produce programs that can be executed by pre-trained vision-language models. Ho…
Visual Question Answering (VQA) Split AVisual Question Answering (VQA) Split BVisual Reasoning