PoTion: Pose MoTion Representation for Action Recognition
Most state-of-the-art methods for action recognition rely on a two-stream architecture that processes appearance and motion independently. In this paper, we claim that considering them jointly offers rich information for action recognition. We introduce a novel representation that gracefully encodes the movement of some semantic keypoints. We use the human joints as these keypoints and term our Pose moTion representation PoTion. Specifically, we first run a state-of-the-art human pose estimator and extract heatmaps for the human joints in each frame. We obtain our PoTion representation by temporally aggregating these probability maps. This is achieved by colorizing each of them depending on the relative time of the frames in the video clip and summing them. This fixed-size representation for an entire video clip is suitable to classify actions using a shallow convolutional neural network. Our experimental evaluation shows that PoTion outperforms other state-of-the-art pose representations. Furthermore, it is complementary to standard appearance and motion streams. When combining PoTion with the recent two-stream I3D approach [5], we obtain state-of-the-art performance on the JHMDB, HMDB and UCF101 datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionSkeleton Based Action RecognitionTemporal Action LocalizationSimilar Papers 제목 키워드 기반
Depth-Aware Action Recognition: Pose-Motion Encoding through Temporal Heatmaps
Most state-of-the-art methods for action recognition rely only on 2D spatial features encoding appearance, motion or pose. However, 2D data lacks the depth information, which is crucial for recognizing fine-grained actio…
Action ClassificationAction RecognitionEXPOTION: Facial Expression and Motion Control for Multimodal Music Generation
We propose Expotion (Facial Expression and Motion Control for Multimodal Music Generation), a generative model leveraging multimodal visual controls - specifically, human facial expressions and upper-body motion - as wel…
parameter-efficient fine-tuningText-to-Music GenerationMachine learning for potion development at Hogwarts
Objective: To determine whether machine learning methods can generate useful potion recipes for research and teaching at Hogwarts School of Witchcraft and Wizardry. Design: Using deep neural networks to classify generate…
Learning multimodal representations for sample-efficient recognition of human actions
Humans interact in rich and diverse ways with the environment. However, the representation of such behavior by artificial agents is often limited. In this work we present \textit{motion concepts}, a novel multimodal repr…
MaCLR: Motion-aware Contrastive Learning of Representations for Videos
We present MaCLR, a novel method to explicitly perform cross-modal self-supervised video representations learning from visual and motion modalities. Compared to previous video representation learning methods that mostly …
Action DetectionAction RecognitionContrastive LearningRepresentation Learning