Multimodal analysis of the predictability of hand-gesture properties
Embodied conversational agents benefit from being able to accompany their speech with gestures. Although many data-driven approaches to gesture generation have been proposed in recent years, it is still unclear whether such systems can consistently generate gestures that convey meaning. We investigate which gesture properties (phase, category, and semantics) can be predicted from speech text and/or audio using contemporary deep learning. In extensive experiments, we show that gesture properties related to gesture meaning (semantics and category) are predictable from text features (time-aligned FastText embeddings) alone, but not from prosodic audio features, while rhythm-related gesture properties (phase) on the other hand can be predicted from audio features better than from text. These results are encouraging as they indicate that it is possible to equip an embodied agent with content-wise meaningful co-speech gestures using a machine-learning model.
Code (0)
등록된 구현이 없습니다.
Tasks
Gesture GenerationRhythmMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Review of Temporal Aspects of Hand Gesture Analysis Applied to Discourse Analysis and Natural Conversation
Lately, there has been an increasing interest in hand gesture analysis systems. Recent works have employed pattern recognition techniques and have focused on the development of systems with more natural user interfaces. …
multimodal interactionSystematic Literature ReviewGesture2Speech: How Far Can Hand Movements Shape Expressive Speech?
Human communication seamlessly integrates speech and bodily motion, where hand gestures naturally complement vocal prosody to express intent, emotion, and emphasis. While recent text-to-speech (TTS) systems have begun in…
Speech SynthesisLexical Retrieval Hypothesis in Multimodal Context
Multimodal corpora have become an essential language resource for language science and grounded natural language processing (NLP) systems due to the growing need to understand and interpret human communication across var…
RetrievalEHWGesture -- A dataset for multimodal understanding of clinical gestures
Hand gesture understanding is essential for several applications in human-computer interaction, including automatic clinical assessment of hand dexterity. While deep learning has advanced static gesture recognition, dyna…
Action Quality AssessmentGesture RecognitionImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input
Human communication combines speech with expressive nonverbal cues such as hand gestures that serve manifold communicative functions. Yet, current generative gesture generation approaches are restricted to simple, repeti…
Gesture Generation