Recognizing Co-Speech Gestures in-the-Wild

2026-05-29 · Sindhu B Hegde, K R Prajwal, Andrew Zisserman arxiv

While humans naturally gesture during speech, only a sparse subset of these movements are visually depictive and semantically linked to specific spoken words. Current multimodal models struggle to capture these semantic co-speech gestures, heavily bottlenecked by a lack of precisely annotated training data. To address this, we introduce the Gesture Recognition in the Wild (GRW) dataset, the first large-scale benchmark designed to map unconstrained human gestures to specific words with frame-accurate temporal boundaries. Comprising 156,688 manually annotated video clips, GRW spans a highly diverse 150-word taxonomy of physical actions, spatial descriptors, and abstract concepts. We leverage GRW to train video models to (a) classify gestures as semantic or not, (b) recognize the word corresponding to a co-speech gesture, and (c) temporally localize the gesture. We also use GRW to establish benchmarks for these three tasks.

📄 PDF Abstract BibTeX arXiv:2605.31589

Code (0)

등록된 구현이 없습니다.

Tasks

Gesture Recognition

Understanding Co-speech Gestures in-the-wild

2025-03-28 · Sindhu B Hegde, K R Prajwal, Taein Kwon, Andrew Zisserman

Co-speech gestures play a vital role in non-verbal communication. In this paper, we introduce a new framework for co-speech gesture understanding in the wild. Specifically, we propose three new tasks and benchmarks to ev…

Active Speaker Detection

Learning Individual Styles of Conversational Gesture

2019-06-10 · CVPR 2019 6 · Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan 외

Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild'' monolo…

Gesture Generation Speech-to-Gesture Translation Translation

Learning Speech-driven 3D Conversational Gestures from Video

2021-02-13 · Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu 외

We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input. Our a…

3D Face Animation Generative Adversarial Network Gesture Generation Hand Pose Estimation+1

Investigating the impact of 2D gesture representation on co-speech gesture generation

2024-06-21 · Teo Guichoux, Laure Soulier, Nicolas Obin, Catherine Pelachaud

Co-speech gestures play a crucial role in the interactions between humans and embodied conversational agents (ECA). Recent deep learning methods enable the generation of realistic, natural co-speech gestures synchronized…

3D Pose Estimation Gesture Generation Pose Estimation

Survey on Emotional Body Gesture Recognition

2018-01-23 · Fatemeh Noroozi, Ciprian Adrian Corneanu, Dorota Kamińska, Tomasz Sapiński 외

Automatic emotion recognition has become a trending research topic in the past decade. While works based on facial expressions or speech abound, recognizing affect from body gestures remains a less explored topic. We pre…

Cultural Vocal Bursts Intensity Prediction Emotion Recognition Gesture Recognition Human Detection+3