VU-BERT: A Unified framework for Visual Dialog
The visual dialog task attempts to train an agent to answer multi-turn questions given an image, which requires the deep understanding of interactions between the image and dialog history. Existing researches tend to employ the modality-specific modules to model the interactions, which might be troublesome to use. To fill in this gap, we propose a unified framework for image-text joint embedding, named VU-BERT, and apply patch projection to obtain vision embedding firstly in visual dialog tasks to simplify the model. The model is trained over two tasks: masked language modeling and next utterance retrieval. These tasks help in learning visual concepts, utterances dependence, and the relationships between these two modalities. Finally, our VU-BERT achieves competitive performance (0.7287 NDCG scores) on VisDial v1.0 Datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMasked Language ModelingRetrievalVisual DialogSimilar Papers 제목 키워드 기반
VD-BERT: A Unified Vision and Dialog Transformer with BERT
Visual dialog is a challenging vision-language task, where a dialog agent needs to answer a series of questions through reasoning on the image content and dialog history. Prior work has mostly focused on various attentio…
Answer GenerationVisual DialogLearning Better Visual Dialog Agents with Pretrained Visual-Linguistic Representation
GuessWhat?! is a two-player visual dialog guessing game where player A asks a sequence of yes/no questions (Questioner) and makes a final guess (Guesser) about a target object in an image, based on answers from player B …
Referring ExpressionReferring Expression ComprehensionVisual DialogVisual GroundingGRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution
Learning from multimodal data has become a popular research topic in recent years. Multimodal coreference resolution (MCR) is an important task in this area. MCR involves resolving the references across different modalit…
coreference-resolutionCoreference ResolutionVisual GroundingUnified Multimodal Model with Unlikelihood Training for Visual Dialog
The task of visual dialog requires a multimodal chatbot to answer sequential questions from humans about image content. Prior work performs the standard likelihood training for answer generation on the positive instances…
Answer GenerationChatbotLanguage ModelingLanguage Modelling+3Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning
Visual Dialog is a challenging vision-language task since the visual dialog agent needs to answer a series of questions after reasoning over both the image content and dialog history. Though existing methods try to deal …
Contrastive LearningQuestion AnsweringVisual DialogVisual Question Answering+1