Delving Deeper into the Decoder for Video Captioning

2020-01-16 · Haoran Chen, Jianmin Li, Xiaolin Hu

Video captioning is an advanced multi-modal task which aims to describe a video clip using a natural language sentence. The encoder-decoder framework is the most popular paradigm for this task in recent years. However, there exist some problems in the decoder of a video captioning model. We make a thorough investigation into the decoder and adopt three techniques to improve the performance of the model. First of all, a combination of variational dropout and layer normalization is embedded into a recurrent unit to alleviate the problem of overfitting. Secondly, a new online method is proposed to evaluate the performance of a model on a validation set so as to select the best checkpoint for testing. Finally, a new training strategy called professional learning is proposed which uses the strengths of a captioning model and bypasses its weaknesses. It is demonstrated in the experiments on Microsoft Research Video Description Corpus (MSVD) and MSR-Video to Text (MSR-VTT) datasets that our model has achieved the best results evaluated by BLEU, CIDEr, METEOR and ROUGE-L metrics with significant gains of up to 18% on MSVD and 3.5% on MSR-VTT compared with the previous state-of-the-art models.

📄 PDF Abstract BibTeX arXiv:2001.05614

Code (1)

WingsBrokenAngel/delving-deeper-into-the-decoder-for-video-captioning 공식 구현 tf

Tasks

DecoderSentenceVideo CaptioningVideo Description

Methods 이 논문이 사용한 방법론

Variational Dropout Variational Dropout is a regularization technique based on dropout, but uses a variational inference grounded approach. In…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Delving Deeper into Convolutional Networks for Learning Video Representations

2015-11-19 · Nicolas Ballas, Li Yao, Chris Pal, Aaron Courville

We propose an approach to learn spatio-temporal features in videos from intermediate visual representations we call "percepts" using Gated-Recurrent-Unit Recurrent Networks (GRUs).Our method relies on percepts that are e…

Action Recognition Decoder Temporal Action Localization Video Captioning

Gated Hierarchical Attention for Image Captioning

2018-10-30 · Qingzhong Wang, Antoni B. Chan

Attention modules connecting encoder and decoders have been widely applied in the field of object recognition, image captioning, visual question answering and neural machine translation, and significantly improves the pe…

Decoder Image Captioning Reinforcement Learning Reinforcement Learning (RL)+2

Video Captioning with Boundary-aware Hierarchical Language Decoding and Joint Video Prediction

2018-07-08 · Xiangxi Shi, Jianfei Cai, Jiuxiang Gu, Shafiq Joty

The explosion of video data on the internet requires effective and efficient technology to generate captions automatically for people who are not able to watch the videos. Despite the great progress of video captioning r…

Decoder Language Modeling Language Modelling Sentence+3

Reconstruction Network for Video Captioning

2018-03-30 · CVPR 2018 6 · Bairui Wang, Lin Ma, Wei zhang, Wei Liu

In this paper, the problem of describing visual contents of a video sequence with natural language is addressed. Unlike previous video captioning work mainly exploiting the cues of video contents to make a language descr…

Decoder Sentence Video Captioning

Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning

2019-06-03 · Wei Zhang, Bairui Wang, Lin Ma, Wei Liu

Decoder reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2