Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs
Dynamic scene graph generation from a video is challenging due to the temporal dynamics of the scene and the inherent temporal fluctuations of predictions. We hypothesize that capturing long-term temporal dependencies is the key to effective generation of dynamic scene graphs. We propose to learn the long-term dependencies in a video by capturing the object-level consistency and inter-object relationship dynamics over object-level long-term tracklets using transformers. Experimental results demonstrate that our Dynamic Scene Graph Detection Transformer (DSG-DETR) outperforms state-of-the-art methods by a significant margin on the benchmark dataset Action Genome. Our ablation studies validate the effectiveness of each component of the proposed approach. The source code is available at https://github.com/Shengyu-Feng/DSG-DETR.
Code (1)
Tasks
Graph GenerationObjectScene Graph DetectionScene Graph GenerationVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies
Recurrent neural networks (RNNs) have achieved state-of-the-art performance on many diverse tasks, from machine translation to surgical activity recognition, yet training RNNs to capture long-term dependencies remains di…
Activity RecognitionMachine TranslationTranslationTraining biologically plausible recurrent neural networks on cognitive tasks with long-term dependencies
Training recurrent neural networks (RNNs) has become a go-to approach for generating and evaluating mechanistic neural hypotheses for cognition. The ease and efficiency of training RNNs with backpropagation through time …
Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory
The effectiveness of recurrent neural networks can be largely influenced by their ability to store into their dynamical memory information extracted from input sequences at different frequencies and timescales. Such a fe…
speech-recognitionSpeech RecognitionReconstructive Sequence-Graph Network for Video Summarization
Exploiting the inner-shot and inter-shot dependencies is essential for key-shot based video summarization. Current approaches mainly devote to modeling the video as a frame sequence by recurrent neural networks. However,…
Video SummarizationBayesian Graph Convolution LSTM for Skeleton Based Action Recognition
We propose a framework for recognizing human actions from skeleton data by modeling the underlying dynamic process that generates the motion pattern. We capture three major factors that contribute to the complexity of th…
Action RecognitionAnatomyBayesian InferenceSkeleton Based Action Recognition