MS$^2$L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition
In this paper, we address self-supervised representation learning from human skeletons for action recognition. Previous methods, which usually learn feature presentations from a single reconstruction task, may come across the overfitting problem, and the features are not generalizable for action recognition. Instead, we propose to integrate multiple tasks to learn more general representations in a self-supervised manner. To realize this goal, we integrate motion prediction, jigsaw puzzle recognition, and contrastive learning to learn skeleton features from different aspects. Skeleton dynamics can be modeled through motion prediction by predicting the future sequence. And temporal patterns, which are critical for action recognition, are learned through solving jigsaw puzzles. We further regularize the feature space by contrastive learning. Besides, we explore different training strategies to utilize the knowledge from self-supervised tasks for action recognition. We evaluate our multi-task self-supervised learning approach with action classifiers trained under different configurations, including unsupervised, semi-supervised and fully-supervised settings. Our experiments on the NW-UCLA, NTU RGB+D, and PKUMMD datasets show remarkable performance for action recognition, demonstrating the superiority of our method in learning more discriminative and general features. Our project website is available at https://langlandslin.github.io/projects/MSL/.
Code (1)
Tasks
Action RecognitionContrastive Learningmotion predictionRepresentation LearningSelf-Supervised LearningSkeleton Based Action RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorization
3D Skeleton-based human action recognition has attracted increasing attention in recent years. Most of the existing work focuses on supervised learning which requires a large number of labeled action sequences that are o…
3D Action RecognitionAction RecognitionColorizationRepresentation Learning+3Skeleton-Contrastive 3D Action Representation Learning
This paper strives for self-supervised learning of a feature space suitable for skeleton-based action recognition. Our proposal is built upon learning invariances to input skeleton representations and various skeleton au…
Action RecognitionContrastive LearningFew-Shot Skeleton-Based Action RecognitionRepresentation Learning+4SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-supervised Skeleton Action Recognition
Fully supervised skeleton-based action recognition has achieved great progress with the blooming of deep learning techniques. However, these methods require sufficient labeled data which is not easy to obtain. In contras…
Action RecognitionDecoderSelf-supervised Skeleton-based Action RecognitionSkeleton Based Action RecognitionHierarchically Self-Supervised Transformer for Human Skeleton Representation Learning
Despite the success of fully-supervised human skeleton sequence modeling, utilizing self-supervised pre-training for skeleton sequence representation learning has been an active field because acquiring task-specific skel…
Action DetectionAction RecognitionContrastive Learningmotion prediction+1ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting
Current self-supervised approaches for skeleton action representation learning often focus on constrained scenarios, where videos and skeleton data are recorded in laboratory settings. When dealing with estimated skeleto…
Action ClassificationAction Recognitionmotion retargetingRepresentation Learning+2