Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modeling
Transformer-based models have achieved top performance on major video recognition benchmarks. Benefiting from the self-attention mechanism, these models show stronger ability of modeling long-range dependencies compared to CNN-based models. However, significant computation overheads, resulted from the quadratic complexity of self-attention on top of a tremendous number of tokens, limit the use of existing video transformers in applications with limited resources like mobile devices. In this paper, we extend Mobile-Former to Video Mobile-Former, which decouples the video architecture into a lightweight 3D-CNNs for local context modeling and a Transformer modules for global interaction modeling in a parallel fashion. To avoid significant computational cost incurred by computing self-attention between the large number of local patches in videos, we propose to use very few global tokens (e.g., 6) for a whole video in Transformers to exchange information with 3D-CNNs with a cross-attention mechanism. Through efficient global spatial-temporal modeling, Video Mobile-Former significantly improves the video recognition performance of alternative lightweight baselines, and outperforms other efficient CNN-based models at the low FLOP regime from 500M to 6G total FLOPs on various video recognition tasks. It is worth noting that Video Mobile-Former is the first Transformer-based video model which constrains the computational budget within 1G FLOPs.
Code (0)
등록된 구현이 없습니다.
Tasks
Video RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Mobile Video Action Recognition
Video action recognition, which is topical in computer vision and video analysis, aims to allocate a short video clip to a pre-defined category such as brushing hair or climbing stairs. Recent works focus on action recog…
Action RecognitionTemporal Action LocalizationVideo Swin Transformer
The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchmarks. These video models are all built o…
Action ClassificationAction RecognitionGeneral ClassificationInductive Bias+3Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognition
Recent video recognition models utilize Transformer models for long-range spatio-temporal context modeling. Video transformer designs are based on self-attention that can model global context at a high computational cost…
Action RecognitionTemporal Action LocalizationVideo RecognitionDualFormer: Local-Global Stratified Transformer for Efficient Video Recognition
While transformers have shown great potential on video recognition with their strong capability of capturing long-range dependencies, they often suffer high computational costs induced by the self-attention to the huge n…
Video RecognitionSpace-time Mixing Attention for Video Transformer
This paper is on video recognition using Transformers. Very recent attempts in this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, signi…
Action ClassificationAction RecognitionAction Recognition In VideosVideo Recognition