On the Importance of Spatial Relations for Few-shot Action Recognition
Deep learning has achieved great success in video recognition, yet still struggles to recognize novel actions when faced with only a few examples. To tackle this challenge, few-shot action recognition methods have been proposed to transfer knowledge from a source dataset to a novel target dataset with only one or a few labeled videos. However, existing methods mainly focus on modeling the temporal relations between the query and support videos while ignoring the spatial relations. In this paper, we find that the spatial misalignment between objects also occurs in videos, notably more common than the temporal inconsistency. We are thus motivated to investigate the importance of spatial relations and propose a more accurate few-shot action recognition method that leverages both spatial and temporal information. Particularly, a novel Spatial Alignment Cross Transformer (SA-CT) which learns to re-adjust the spatial relations and incorporates the temporal information is contributed. Experiments reveal that, even without using any temporal information, the performance of SA-CT is comparable to temporal based methods on 3/4 benchmarks. To further incorporate the temporal information, we propose a simple yet effective Temporal Mixer module. The Temporal Mixer enhances the video representation and improves the performance of the full SA-CT model, achieving very competitive results. In this work, we also exploit large-scale pretrained models for few-shot action recognition, providing useful insights for this research direction.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionFew-Shot action recognitionFew Shot Action RecognitionVideo RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Revisiting the Spatial and Temporal Modeling for Few-shot Action Recognition
Spatial and temporal modeling is one of the most core aspects of few-shot action recognition. Most previous works mainly focus on long-term temporal relation modeling based on high-level spatial representations, without …
Action RecognitionFew-Shot action recognitionFew Shot Action RecognitionSqueeze-and-Excitation on Spatial and Temporal Deep Feature Space for Action Recognition
Spatial and temporal features are two key and complementary information for human action recognition. In order to make full use of the intra-frame spatial characteristics and inter-frame temporal relationships, we propos…
Action RecognitionTemporal Action LocalizationCLTA: Contents and Length-based Temporal Attention for Few-shot Action Recognition
Few-shot action recognition has attracted increasing attention due to the difficulty in acquiring the properly labelled training samples. Current works have shown that preserving spatial information and comparing video d…
Action RecognitionFew-Shot action recognitionFew Shot Action RecognitionFew-shot Action Recognition via Intra- and Inter-Video Information Maximization
Current few-shot action recognition involves two primary sources of information for classification:(1) intra-video information, determined by frame content within a single video clip, and (2) inter-video information, mea…
Action RecognitionFew-Shot action recognitionFew Shot Action RecognitionTemporal Action Localization+1Video-to-Task Learning via Motion-Guided Attention for Few-Shot Action Recognition
In recent years, few-shot action recognition has achieved remarkable performance through spatio-temporal relation modeling. Although a wide range of spatial and temporal alignment modules have been proposed, they primari…
Action RecognitionFew-Shot action recognitionFew Shot Action Recognition