Video Fill in the Blank with Merging LSTMs
Given a video and its incomplete textural description with missing words, the Video-Fill-in-the-Blank (ViFitB) task is to automatically find the missing word. The contextual information of the sentences are important to infer the missing words; the visual cues are even more crucial to get a more accurate inference. In this paper, we presents a new method which intuitively takes advantage of the structure of the sentences and employs merging LSTMs (to merge two LSTMs) to tackle the problem with embedded textural and visual cues. In the experiments, we have demonstrated the superior performance of the proposed method on the challenging "Movie Fill-in-the-Blank" dataset.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Video Fill In the Blank using LR/RL LSTMs with Spatial-Temporal Attentions
Given a video and a description sentence with one missing word (we call it the "source sentence"), Video-Fill-In-the-Blank (VFIB) problem is to find the missing word automatically. The contextual information of the sente…
SentenceFill-in-the-Blank: A Challenging Video Understanding Evaluation Framework
We propose fill-in-the-blanks as a video understanding evaluation framework. The task tests a model's understanding of a video by requiring the model to predict a masked noun phrase in the caption of the video, given the…
Multiple-choiceQuestion AnsweringVideo CaptioningVideo Question Answering+1FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework
We propose fill-in-the-blanks as a video understanding evaluation framework and introduce FIBER -- a novel dataset consisting of 28,000 videos and descriptions in support of this evaluation framework. The fill-in-the-bla…
Language ModellingMultiple-choiceQuestion AnsweringVideo Captioning+2A-TIP: Attribute-aware Text Infilling via Pre-trained Language Model
Text infilling aims to restore incomplete texts by filling in blanks, which has attracted more attention recently because of its wide application in ancient text restoration and text rewriting. However, attribute- aware …
Ancient Text RestorationAttributeLanguage ModelingLanguage Modelling+1Blank Language Models
We propose Blank Language Model (BLM), a model that generates sequences by dynamically creating and filling in blanks. The blanks control which part of the sequence to expand, making BLM ideal for a variety of text editi…
Ancient Text RestorationLanguage ModelingLanguage ModellingStyle Transfer