LM2D: Lyrics- and Music-Driven Dance Synthesis
Dance typically involves professional choreography with complex movements that follow a musical rhythm and can also be influenced by lyrical content. The integration of lyrics in addition to the auditory dimension, enriches the foundational tone and makes motion generation more amenable to its semantic meanings. However, existing dance synthesis methods tend to model motions only conditioned on audio signals. In this work, we make two contributions to bridge this gap. First, we propose LM2D, a novel probabilistic architecture that incorporates a multimodal diffusion model with consistency distillation, designed to create dance conditioned on both music and lyrics in one diffusion generation step. Second, we introduce the first 3D dance-motion dataset that encompasses both music and lyrics, obtained with pose estimation technologies. We evaluate our model against music-only baseline models with objective metrics and human evaluations, including dancers and choreographers. The results demonstrate LM2D is able to produce realistic and diverse dance matching both lyrics and music. A video summary can be accessed at: https://youtu.be/4XCgvYookvA.
Code (0)
등록된 구현이 없습니다.
Tasks
Motion GenerationPose EstimationRhythmMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Music- and Lyrics-driven Dance Synthesis
Lyrics often convey information about the songs that are beyond the auditory dimension, enriching the semantic meaning of movements and musical themes. Such insights are important in the dance choreography domain. Howeve…
TripletEnabling Embodied Analogies in Intelligent Music Systems
The present methodology is aimed at cross-modal machine learning and uses multidisciplinary tools and methods drawn from a broad range of areas and disciplines, including music, systematic musicology, dance, motion captu…
Audio Signal ProcessingBIG-bench Machine LearningA Survey on Recent Deep Learning-driven Singing Voice Synthesis Systems
Singing voice synthesis (SVS) is a task that aims to generate audio signals according to musical scores and lyrics. With its multifaceted nature concerning music and language, producing singing voices indistinguishable f…
Deep LearningSinging Voice SynthesisLyrics Matter: Exploiting the Power of Learnt Representations for Music Popularity Prediction
Accurately predicting music popularity is a critical challenge in the music industry, offering benefits to artists, producers, and streaming platforms. Prior research has largely focused on audio features, social metadat…
Music-driven Dance Regeneration with Controllable Key Pose Constraints
In this paper, we propose a novel framework for music-driven dance motion synthesis with controllable key pose constraint. In contrast to methods that generate dance motion sequences only based on music without any other…
DecoderMotion Synthesis