Modeling the Relative Visual Tempo for Self-supervised Skeleton-based Action Recognition
Visual tempo characterizes the dynamics and the temporal evolution, which helps describe actions. Recent approaches directly perform visual tempo prediction on skeleton sequences, which may suffer from insufficient feature representation issue. In this paper, we observe that relative visual tempo is more in line with human intuition, and thus providing more effective supervision signals. Based on this, we propose a novel Relative Visual Tempo Contrastive Learning framework for skeleton action Representation (RVTCLR). Specifically, we design a Relative Visual Tempo Learning (RVTL) task to explore the motion information in intra-video clips, and an Appearance-Consistency (AC) task to learn appearance information simultaneously, resulting in more representative spatiotemporal features. Furthermore, skeleton sequence data is much sparser than RGB data, making the network learn shortcuts, and overfit to low-level information such as skeleton scales. To learn high-order semantics, we further design a new Distribution-Consistency (DC) branch, containing three components: Skeleton-specific Data Augmentation (SDA), Fine-grained Skeleton Encoding Module (FSEM), and Distribution-aware Diversity (DD) Loss. We term our entire method (RVTCLR with DC) as RVTCLR+. Extensive experiments on NTU RGB+D 60 and NTU RGB+D 120 datasets demonstrate that our RVTCLR+ can achieve competitive results over the state-of-the-art methods. Code is available at https://github.com/Zhuysheng/RVTCLR.
Code (1)
Tasks
Action RecognitionContrastive LearningData AugmentationSelf-Supervised Human Action RecognitionSelf-supervised Skeleton-based Action RecognitionSkeleton Based Action RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Inter-slice Super-resolution of Magnetic Resonance Images by Pre-training and Self-supervised Fine-tuning
In clinical practice, 2D magnetic resonance (MR) sequences are widely adopted. While individual 2D slices can be stacked to form a 3D volume, the relatively large slice spacing can pose challenges for both image visualiz…
Super-ResolutionMasked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning
Benefiting from masked visual modeling, self-supervised video representation learning has achieved remarkable progress. However, existing methods focus on learning representations from scratch through reconstructing low-…
Action ClassificationAction RecognitionRepresentation LearningSelf-Supervised Action RecognitionUnsupervised Learning Optical Flow in Multi-frame Dynamic Environment Using Temporal Dynamic Modeling
For visual estimation of optical flow, a crucial function for many vision tasks, unsupervised learning, using the supervision of view synthesis has emerged as a promising alternative to supervised methods, since ground-t…
Optical Flow EstimationLearning Monocular Visual Odometry via Self-Supervised Long-Term Modeling
Monocular visual odometry (VO) suffers severely from error accumulation during frame-to-frame pose estimation. In this paper, we present a self-supervised learning method for VO with special consideration for consistency…
GPUMonocular Visual OdometryPose EstimationPose Prediction+2The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning
Contrastive learning of auditory and visual perception has been extremely successful when investigated individually. However, there are still major questions on how we could integrate principles learned from both domains…
Contrastive LearningRepresentation LearningSelf-Supervised Learning