Flow-Distilled IP Two-Stream Networks for Compressed Video Action Recognition
Two-stream networks have achieved great success in video recognition. A two-stream network combines a spatial stream of RGB frames and a temporal stream of Optical Flow to make predictions. However, the temporal redundancy of RGB frames as well as the high-cost of optical flow computation creates challenges for both the performance and efficiency. Recent works instead use modern compressed video modalities as an alternative to the RGB spatial stream and improve the inference speed by orders of magnitudes. Previous works create one stream for each modality which are combined with an additional temporal stream through late fusion. This is redundant since some modalities like motion vectors already contain temporal information. Based on this observation, we propose a compressed domain two-stream network IP TSN for compressed video recognition, where the two streams are represented by the two types of frames (I and P frames) in compressed videos, without needing a separate temporal stream. With this goal, we propose to fully exploit the motion information of P-stream through generalized distillation from optical flow, which largely improves the efficiency and accuracy. Our P-stream runs 60 times faster than using optical flow while achieving higher accuracy. Our full IP TSN, evaluated over public action recognition benchmarks (UCF101, HMDB51 and a subset of Kinetics), outperforms other compressed domain methods by large margins while improving the total inference speed by 20%.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionOptical Flow EstimationTemporal Action LocalizationVideo RecognitionVocal Bursts Valence PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
D3D: Distilled 3D Networks for Video Action Recognition
State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input. In recent w…
Action ClassificationAction RecognitionOptical Flow EstimationTemporal Action LocalizationMVFlow: Deep Optical Flow Estimation of Compressed Videos with Motion Vector Prior
In recent years, many deep learning-based methods have been proposed to tackle the problem of optical flow estimation and achieved promising results. However, they hardly consider that most videos are compressed and thus…
Optical Flow EstimationDMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition
Motion has shown to be useful for video understanding, where motion is typically represented by optical flow. However, computing flow from video frames is very time-consuming. Recent works directly leverage the motion ve…
Action ClassificationAction RecognitionAction Recognition In VideosOptical Flow Estimation+2Video Classification With CNNs: Using The Codec As A Spatio-Temporal Activity Sensor
We investigate video classification via a two-stream convolutional neural network (CNN) design that directly ingests information extracted from compressed video bitstreams. Our approach begins with the observation that a…
ClassificationCloud ComputingCPUGeneral Classification+3Compressed Video Action Recognition with Refined Motion Vector
Although CNN has reached satisfactory performance in image-related tasks, using CNN to process videos is much more challenging due to the enormous size of raw video streams. In this work, we propose to use motion vectors…
Action RecognitionOptical Flow EstimationTemporal Action LocalizationVideo Compression