Self-Supervised 3D Action Representation Learning with Skeleton Cloud Colorization
3D Skeleton-based human action recognition has attracted increasing attention in recent years. Most of the existing work focuses on supervised learning which requires a large number of labeled action sequences that are often expensive and time-consuming to annotate. In this paper, we address self-supervised 3D action representation learning for skeleton-based action recognition. We investigate self-supervised representation learning and design a novel skeleton cloud colorization technique that is capable of learning spatial and temporal skeleton representations from unlabeled skeleton sequence data. We represent a skeleton action sequence as a 3D skeleton cloud and colorize each point in the cloud according to its temporal and spatial orders in the original (unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud, we design an auto-encoder framework that can learn spatial-temporal features from the artificial color labels of skeleton joints effectively. Specifically, we design a two-steam pretraining network that leverages fine-grained and coarse-grained colorization to learn multi-scale spatial-temporal features. In addition, we design a Masked Skeleton Cloud Repainting task that can pretrain the designed auto-encoder framework to learn informative representations. We evaluate our skeleton cloud colorization approach with linear classifiers trained under different configurations, including unsupervised, semi-supervised, fully-supervised, and transfer learning settings. Extensive experiments on NTU RGB+D, NTU RGB+D 120, PKU-MMD, NW-UCLA, and UWA3D datasets show that the proposed method outperforms existing unsupervised and semi-supervised 3D action recognition methods by large margins and achieves competitive performance in supervised 3D action recognition as well.
Code (0)
등록된 구현이 없습니다.
Tasks
3D Action RecognitionAction RecognitionColorizationRepresentation LearningSkeleton Based Action RecognitionTemporal Action LocalizationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning
Skeleton-based human action recognition has attracted increasing attention in recent years. However, most of the existing works focus on supervised learning which requiring a large number of annotated action sequences th…
3D Action RecognitionAction RecognitionColorizationRepresentation Learning+1Contrastive Self-Supervised Learning for Skeleton Representations
Human skeleton point clouds are commonly used to automatically classify and predict the behaviour of others. In this paper, we use a contrastive self-supervised learning method, SimCLR, to learn representations that capt…
Graph Neural Networkmotion predictionSelf-Supervised LearningSkeleton-Contrastive 3D Action Representation Learning
This paper strives for self-supervised learning of a feature space suitable for skeleton-based action recognition. Our proposal is built upon learning invariances to input skeleton representations and various skeleton au…
Action RecognitionContrastive LearningFew-Shot Skeleton-Based Action RecognitionRepresentation Learning+4ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting
Current self-supervised approaches for skeleton action representation learning often focus on constrained scenarios, where videos and skeleton data are recorded in laboratory settings. When dealing with estimated skeleto…
Action ClassificationAction Recognitionmotion retargetingRepresentation Learning+2Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning
Despite the success of fully-supervised human skeleton sequence modeling, utilizing self-supervised pre-training for skeleton sequence representation learning has been an active field because acquiring task-specific skel…
Action DetectionAction RecognitionContrastive Learningmotion prediction+1