Heterogeneous Skeleton-Based Action Representation Learning
Skeleton-based human action recognition has received widespread attention in recent years due to its diverse range of application scenarios. Due to the different sources of human skeletons, skeleton data naturally exhibit heterogeneity. The previous works, however, overlook the heterogeneity of human skeletons and solely construct models tailored for homogeneous skeletons. This work addresses the challenge of heterogeneous skeleton-based action representation learning, specifically focusing on processing skeleton data that varies in joint dimensions and topological structures. The proposed framework comprises two primary components: heterogeneous skeleton processing and unified representation learning. The former first converts two-dimensional skeleton data into three-dimensional skeleton via an auxiliary network, and then constructs a prompted unified skeleton using skeleton-specific prompts. We also design an additional modality named semantic motion encoding to harness the semantic information within skeletons. The latter module learns a unified action representation using a shared backbone network that processes different heterogeneous skeletons. Extensive experiments on the NTU-60, NTU-120, and PKU-MMD II datasets demonstrate the effectiveness of our method in various tasks of action understanding. Our approach can be applied to action recognition in robots with different humanoid structures.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionAction UnderstandingRepresentation LearningTemporal Action LocalizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabularies
Skeleton data used for action recognition are acquired from a wide range of sources, including depth sensors, marker-based motion capture systems, and 2D/3D pose estimators. These sources yield skeletons that differ in j…
Contrastive LearningAction RecognitionSkeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models
Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including cross-modal and multi-modal scenarios. However, their direct applicatio…
Representation LearningAction RecognitionUniversal Skeleton Understanding via Differentiable Rendering and MLLMs
Multimodal large language models (MLLMs) exhibit strong visual-language reasoning, yet cannot process structured, non-visual data such as human skeletons. Existing methods either compress skeleton dynamics into lossy fea…
Action RecognitionQuestion AnsweringMotion CaptioningEnhancing Action Recognition from Low-Quality Skeleton Data via Part-Level Knowledge Distillation
Skeleton-based action recognition is vital for comprehending human-centric videos and has applications in diverse domains. One of the challenges of skeleton-based action recognition is dealing with low-quality data, such…
Action RecognitionGeneral KnowledgeKnowledge DistillationSkeleton Based Action Recognition+1Skeleton-Contrastive 3D Action Representation Learning
This paper strives for self-supervised learning of a feature space suitable for skeleton-based action recognition. Our proposal is built upon learning invariances to input skeleton representations and various skeleton au…
Action RecognitionContrastive LearningFew-Shot Skeleton-Based Action RecognitionRepresentation Learning+4