Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
Localizing persons and recognizing their actions from videos is a challenging task towards high-level video understanding. Recent advances have been achieved by modeling direct pairwise relations between entities. In this paper, we take one step further, not only model direct relations between pairs but also take into account indirect higher-order relations established upon multiple elements. We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context. To this end, we design an Actor-Context-Actor Relation Network (ACAR-Net) which builds upon a novel High-order Relation Reasoning Operator and an Actor-Context Feature Bank to enable indirect relation reasoning for spatio-temporal action localization. Experiments on AVA and UCF101-24 datasets show the advantages of modeling actor-context-actor relations, and visualization of attention maps further verifies that our model is capable of finding relevant higher-order relations to support action detection. Notably, our method ranks first in the AVA-Kineticsaction localization task of ActivityNet Challenge 2020, out-performing other entries by a significant margin (+6.71mAP). Training code and models will be available at https://github.com/Siyu-C/ACAR-Net.
Code (3)
Tasks
Action DetectionAction LocalizationAction RecognitionRelationRelation NetworkSpatio-Temporal Action LocalizationTemporal Action LocalizationVideo UnderstandingSimilar Papers 제목 키워드 기반
Spatio-Temporal Context for Action Detection
Research in action detection has grown in the recentyears, as it plays a key role in video understanding. Modelling the interactions (either spatial or temporal) between actors and their context has proven to be essentia…
Action DetectionVideo UnderstandingActor-Centric Relation Network
Current state-of-the-art approaches for spatio-temporal action localization rely on detections at the frame level and model temporal context with 3D ConvNets. Here, we go one step further and model spatio-temporal relati…
Action ClassificationAction DetectionAction LocalizationAction Recognition+4MRSN: Multi-Relation Support Network for Video Action Detection
Action detection is a challenging video understanding task, requiring modeling spatio-temporal and interaction relations. Current methods usually model actor-actor and actor-context relations separately, ignoring their c…
Action DetectionRelationVideo Action DetectionVideo UnderstandingstagNet: An Attentive Semantic RNN for Group Activity Recognition
Group activity recognition plays a fundamental role in a variety of applications, e.g. sports video analysis and intelligent surveillance. How to model the spatio-temporal contextual information in a scene still remains …
Activity RecognitionGroup Activity RecognitionTemporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis
Multimodal Sentiment Analysis integrates Linguistic, Visual, and Acoustic. Mainstream approaches based on modality-invariant and modality-specific factorization or on complex fusion still rely on spatiotemporal mixed mod…
Multimodal Sentiment AnalysisRepresentation Learning