DanceFusion: A Spatio-Temporal Skeleton Diffusion Transformer for Audio-Driven Dance Motion Reconstruction
This paper introduces DanceFusion, a novel framework for reconstructing and generating dance movements synchronized to music, utilizing a Spatio-Temporal Skeleton Diffusion Transformer. The framework adeptly handles incomplete and noisy skeletal data common in short-form dance videos on social media platforms like TikTok. DanceFusion incorporates a hierarchical Transformer-based Variational Autoencoder (VAE) integrated with a diffusion model, significantly enhancing motion realism and accuracy. Our approach introduces sophisticated masking techniques and a unique iterative diffusion process that refines the motion sequences, ensuring high fidelity in both motion generation and synchronization with accompanying audio cues. Comprehensive evaluations demonstrate that DanceFusion surpasses existing methods, providing state-of-the-art performance in generating dynamic, realistic, and stylistically diverse dance motions. Potential applications of this framework extend to content creation, virtual reality, and interactive entertainment, promising substantial advancements in automated dance generation. Visit our project page at https://th-mlab.github.io/DanceFusion/.
Code (0)
등록된 구현이 없습니다.
Tasks
Motion GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition
In action recognition, although the combination of spatio-temporal videos and skeleton features can improve the recognition performance, a separate model and balancing feature representation for cross-modal data are requ…
Action RecognitionDecoderTemporal Action LocalizationHierarchical Graph Convolutional Skeleton Transformer for Action Recognition
Graph convolutional networks (GCNs) have emerged as dominant methods for skeleton-based action recognition. However, they still suffer from two problems, namely, neighborhood constraints and entangled spatiotemporal feat…
Action RecognitionDisentanglementSkeleton Based Action RecognitionDD-GCN: Directed Diffusion Graph Convolutional Network for Skeleton-based Human Action Recognition
Graph Convolutional Networks (GCNs) have been widely used in skeleton-based human action recognition. In GCN-based methods, the spatio-temporal graph is fundamental for capturing motion patterns. However, existing approa…
Action RecognitionTemporal Action LocalizationSpatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition
Capturing the dependencies between joints is critical in skeleton-based action recognition task. Transformer shows great potential to model the correlation of important joints. However, the existing Transformer-based met…
Action RecognitionSkeleton Based Action RecognitionTemporal Decoupling Graph Convolutional Network for Skeleton-based Gesture Recognition
Skeleton-based gesture recognition methods have achieved high success using Graph Convolutional Network (GCN), which commonly uses an adjacency matrix to model the spatial topology of skeletons. However, previous methods…
Gesture RecognitionHand Gesture RecognitionSkeleton Based Action Recognition