Excitation Backprop for RNNs
Deep models are state-of-the-art for many vision tasks including video action recognition and video captioning. Models are trained to caption or classify activity in videos, but little is known about the evidence used to make such decisions. Grounding decisions made by deep networks has been studied in spatial visual content, giving more insight into model predictions for images. However, such studies are relatively lacking for models of spatiotemporal visual content - videos. In this work, we devise a formulation that simultaneously grounds evidence in space and time, in a single pass, using top-down saliency. We visualize the spatiotemporal cues that contribute to a deep model's classification/captioning output using the model's internal representation. Based on these spatiotemporal cues, we are able to localize segments within a video that correspond with a specific action, or phrase from a caption, without explicitly optimizing/training for these tasks.
Code (1)
Tasks
Action RecognitionTemporal Action LocalizationVideo CaptioningSimilar Papers 제목 키워드 기반
A Gentle Tutorial of Recurrent Neural Network with Error Backpropagation
We describe recurrent neural networks (RNNs), which have attracted great attention on sequential tasks, such as handwriting recognition, speech recognition and image to text. However, compared to general feedforward neur…
Handwriting RecognitionImage to textspeech-recognitionSpeech RecognitionLearning Longer-term Dependencies in RNNs with Auxiliary Losses
Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is diffi…
Document ClassificationGeneral Classificationimage-classificationImage ClassificationLearning Longer-term Dependencies in RNNs with Auxiliary Losses
Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is …
Document ClassificationGeneral Classificationimage-classificationImage ClassificationTop-down Neural Attention by Excitation Backprop
We aim to model the top-down attention of a Convolutional Neural Network (CNN) classifier for generating task-specific attention maps. Inspired by a top-down human visual attention model, we propose a new backpropagation…
Backpropagation Through Time For Networks With Long-Term Dependencies
Backpropagation through time (BPTT) is a technique of updating tuned parameters within recurrent neural networks (RNNs). Several attempts at creating such an algorithm have been made including: Nth Ordered Approximations…