End-to-end Dense Video Captioning as Sequence Generation
Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event. Previous approaches usually follow a two-stage generative process, which first proposes a segment for each event, then renders a caption for each identified segment. Recent advances in large-scale sequence generation pretraining have seen great success in unifying task formulation for a great variety of tasks, but so far, more complex tasks such as dense video captioning are not able to fully utilize this powerful paradigm. In this work, we show how to model the two subtasks of dense video captioning jointly as one sequence generation task, and simultaneously predict the events and the corresponding descriptions. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks such as end-to-end dense video captioning integrated into large-scale pretrained models.
Code (0)
등록된 구현이 없습니다.
Tasks
Dense Video CaptioningDescriptiveVideo CaptioningSimilar Papers 제목 키워드 기반
End-to-end Dense Video Captioning as Sequence Generation
Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event. Previous approaches usually follow a two-stage generative process, which first proposes …
Dense Video CaptioningDescriptiveVideo CaptioningTeam RUC_AIM3 Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense Video Captioning
Detecting meaningful events in an untrimmed video is essential for dense video captioning. In this work, we propose a novel and simple model for event sequence generation and explore temporal relationships of the event s…
Dense CaptioningDense Video CaptioningTask 2Video CaptioningBest Vision Technologies Submission to ActivityNet Challenge 2018-Task: Dense-Captioning Events in Videos
This note describes the details of our solution to the dense-captioning events in videos task of ActivityNet Challenge 2018. Specifically, we solve this problem with a two-stage way, i.e., first temporal event proposal a…
Dense CaptioningOptical Flow EstimationSentenceVideo CaptioningStreamlined Dense Video Captioning
Dense video captioning is an extremely challenging task since accurate and coherent description of events in a video requires holistic understanding of video contents as well as contextual reasoning of individual events.…
Dense Video CaptioningReinforcement LearningVideo CaptioningA Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer
Dense video captioning aims to localize and describe important events in untrimmed videos. Existing methods mainly tackle this task by exploiting only visual features, while completely neglecting the audio track. Only a …
Dense Video CaptioningTemporal Action Proposal GenerationVideo Captioning