IF-TTN: Information Fused Temporal Transformation Network for Video Action Recognition
Effective spatiotemporal feature representation is crucial to the video-based action recognition task. Focusing on discriminate spatiotemporal feature learning, we propose Information Fused Temporal Transformation Network (IF-TTN) for action recognition on top of popular Temporal Segment Network (TSN) framework. In the network, Information Fusion Module (IFM) is designed to fuse the appearance and motion features at multiple ConvNet levels for each video snippet, forming a short-term video descriptor. With fused features as inputs, Temporal Transformation Networks (TTN) are employed to model middle-term temporal transformation between the neighboring snippets following a sequential order. As TSN itself depicts long-term temporal structure by segmental consensus, the proposed network comprehensively considers multiple granularity temporal features. Our IF-TTN achieves the state-of-the-art results on two most popular action recognition datasets: UCF101 and HMDB51. Empirical investigation reveals that our architecture is robust to the input motion map quality. Replacing optical flow with the motion vectors from compressed video stream, the performance is still comparable to the flow-based methods while the testing speed is 10x faster.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionOptical Flow EstimationTemporal Action LocalizationSimilar Papers 제목 키워드 기반
Temporal Deformable Residual Networks for Action Segmentation in Videos
This paper is about temporal segmentation of human actions in videos. We introduce a new model -- temporal deformable residual network (TDRN) -- aimed at analyzing video intervals at multiple temporal scales for labelin…
Action SegmentationSegmentationModelling Temporal Information Using Discrete Fourier Transform for Video Classification
Recently, video classification attracts intensive research efforts. However, most existing works are based on framelevel visual features, which might fail to model the temporal information, e.g. characteristics accumulat…
Action RecognitionClassificationEmotion ClassificationGeneral Classification+2Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition
Acquiring spatio-temporal states of an action is the most crucial step for action classification. In this paper, we propose a data level fusion strategy, Motion Fused Frames (MFFs), designed to fuse motion information in…
Action ClassificationGeneral ClassificationGesture RecognitionHand Gesture Recognition+1Consistent and Editable: A Balanced Framework for Text-Guided Video Editing
Recently, diffusion models have achieved considerable success in the text-guided video editing domain. However, existing works often struggle to balance the trade-off between temporal consistency and editability in video…
Video 3D Sampling for Self-supervised Representation Learning
Most of the existing video self-supervised methods mainly leverage temporal signals of videos, ignoring that the semantics of moving objects and environmental information are all critical for video-related tasks. In this…
Action RecognitionRepresentation LearningRetrievalVideo Retrieval