Generating by Understanding: Neural Visual Generation with Logical Symbol Groundings
Making neural visual generative models controllable by logical reasoning systems is promising for improving faithfulness, transparency, and generalizability. We propose the Abductive visual Generation (AbdGen) approach to build such logic-integrated models. A vector-quantized symbol grounding mechanism and the corresponding disentanglement training method are introduced to enhance the controllability of logical symbols over generation. Furthermore, we propose two logical abduction methods to make our approach require few labeled training data and support the induction of latent logical generative rules from data. We experimentally show that our approach can be utilized to integrate various neural generative models with logical reasoning systems, by both learning from scratch or utilizing pre-trained models directly. The code is released at https://github.com/future-item/AbdGen.
Code (2)
Tasks
DisentanglementLogical ReasoningSimilar Papers 제목 키워드 기반
V-LoL: A Diagnostic Dataset for Visual Logical Learning
Despite the successes of recent developments in visual AI, different shortcomings still exist; from missing exact logical reasoning, to abstract generalization abilities, to understanding complex and noisy scenes. Unfort…
DiagnosticLogical ReasoningVisual ReasoningLogically Consistent Language Models via Neuro-Symbolic Integration
Large language models (LLMs) are a promising venue for natural language understanding and generation. However, current LLMs are far from reliable: they are prone to generating non-factual information and, more crucially,…
Natural Language UnderstandingGenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles
We challenge text-to-image models with generating escape room puzzle images that are visually appealing, logically solid, and intellectually stimulating. While base image models struggle with spatial relationships and af…
Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning
Human reasoning can be understood as a cooperation between the intuitive, associative "System-1" and the deliberative, logical "System-2". For existing System-1-like methods in visual activity understanding, it is crucia…
Bringing The Consistency Gap: Explicit Structured Memory for Interleaved Image-Text Generation
Existing Vision Language Models (VLMs) often struggle to preserve logic, entity identity, and artistic style during extended, interleaved image-text interactions. We identify this limitation as "Multimodal Context Drift"…
multimodal generationText Generation