GoG: Relation-aware Graph-over-Graph Network for Visual Dialog
Visual dialog, which aims to hold a meaningful conversation with humans about a given image, is a challenging task that requires models to reason the complex dependencies among visual content, dialog history, and current questions. Graph neural networks are recently applied to model the implicit relations between objects in an image or dialog. However, they neglect the importance of 1) coreference relations among dialog history and dependency relations between words for the question representation; and 2) the representation of the image based on the fully represented question. Therefore, we propose a novel relation-aware graph-over-graph network (GoG) for visual dialog. Specifically, GoG consists of three sequential graphs: 1) H-Graph, which aims to capture coreference relations among dialog history; 2) History-aware Q-Graph, which aims to fully understand the question through capturing dependency relations between words based on coreference resolution on the dialog history; and 3) Question-aware I-Graph, which aims to capture the relations between objects in an image based on fully question representation. As an additional feature representation module, we add GoG to the existing visual dialogue model. Experimental results show that our model outperforms the strong baseline in both generative and discriminative settings by a significant margin.
Code (0)
등록된 구현이 없습니다.
Tasks
coreference-resolutionCoreference ResolutionImplicit RelationsRelationVisual DialogSimilar Papers 제목 키워드 기반
Iterative Context-Aware Graph Inference for Visual Dialog
Visual dialog is a challenging task that requires the comprehension of the semantic dependencies among implicit visual and textual contexts. This task can refer to the relation inference in a graphical model with sparse …
Graph AttentionGraph EmbeddingRelationVisual DialogRelation-Aware Graph Attention Network for Visual Question Answering
In order to answer semantically-complicated questions about an image, a Visual Question Answering (VQA) model needs to fully understand the visual scene in the image, especially the interactive dynamics between different…
Graph AttentionImplicit RelationsQuestion AnsweringRelation+2Exploring Sparse Spatial Relation in Graph Inference for Text-Based VQA
Text-based visual question answering (TextVQA) faces the significant challenge of avoiding redundant relational inference. To be specific, a large number of detected objects and optical character recognition (OCR) tokens…
Graph LearningObjectOptical Character RecognitionOptical Character Recognition (OCR)+5Class-level Structural Relation Modelling and Smoothing for Visual Representation Learning
Representation learning for images has been advanced by recent progress in more complex neural models such as the Vision Transformers and new learning theories such as the structural causal models. However, these models …
Graph SamplingRelationRepresentation LearningUnbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network
Recent scene graph generation (SGG) frameworks have focused on learning complex relationships among multiple objects in an image. Thanks to the nature of the message passing neural network (MPNN) that models high-order i…
Graph GenerationRelationRepresentation LearningScene Graph Generation