Semantics-Aware Human Motion Generation from Audio Instructions
Recent advances in interactive technologies have highlighted the prominence of audio signals for semantic encoding. This paper explores a new task, where audio signals are used as conditioning inputs to generate motions that align with the semantics of the audio. Unlike text-based interactions, audio provides a more natural and intuitive communication method. However, existing methods typically focus on matching motions with music or speech rhythms, which often results in a weak connection between the semantics of the audio and generated motions. We propose an end-to-end framework using a masked generative transformer, enhanced by a memory-retrieval attention module to handle sparse and lengthy audio inputs. Additionally, we enrich existing datasets by converting descriptions into conversational style and generating corresponding audio with varied speaker identities. Experiments demonstrate the effectiveness and efficiency of the proposed framework, demonstrating that audio instructions can convey semantics similar to text while providing more practical and user-friendly interactions.
Code (0)
등록된 구현이 없습니다.
Tasks
Motion GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation
Animating high-fidelity video portrait with speech audio is crucial for virtual reality and digital entertainment. While most previous studies rely on accurate explicit structural information, recent works explore the im…
NeRFSemantic Differentiation in Speech Emotion Recognition: Insights from Descriptive and Expressive Speech Roles
Speech Emotion Recognition (SER) is essential for improving human-computer interaction, yet its accuracy remains constrained by the complexity of emotional nuances in speech. In this study, we distinguish between descrip…
Speech Emotion RecognitionSIG-Chat: Spatial Intent-Guided Conversational Gesture Generation Involving How, When and Where
The accompanying actions and gestures in dialogue are often closely linked to interactions with the environment, such as looking toward the interlocutor or using gestures to point to the described target at appropriate m…
Gesture GenerationHigh-fidelity Generalized Emotional Talking Face Generation with Multi-modal Emotion Space Learning
Recently, emotional talking face generation has received considerable attention. However, existing methods only adopt one-hot coding, image, or audio as emotion conditions, thus lacking flexible control in practical appl…
Face GenerationTalking Face GenerationComparison and Analysis of Deep Audio Embeddings for Music Emotion Recognition
Emotion is a complicated notion present in music that is hard to capture even with fine-tuned feature engineering. In this paper, we investigate the utility of state-of-the-art pre-trained deep audio embedding methods to…
Emotion RecognitionFeature EngineeringMusic Emotion Recognition