DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speech

2025-03-21 · Yongkang Cheng, Shaoli Huang, Xuelin Chen, Jifeng Ning, Mingming Gong

Diffusion models have demonstrated remarkable synthesis quality and diversity in generating co-speech gestures. However, the computationally intensive sampling steps associated with diffusion models hinder their practicality in real-world applications. Hence, we present DIDiffGes, for a Decoupled Semi-Implicit Diffusion model-based framework, that can synthesize high-quality, expressive gestures from speech using only a few sampling steps. Our approach leverages Generative Adversarial Networks (GANs) to enable large-step sampling for diffusion model. We decouple gesture data into body and hands distributions and further decompose them into marginal and conditional distributions. GANs model the marginal distribution implicitly, while L2 reconstruction loss learns the conditional distributions exciplictly. This strategy enhances GAN training stability and ensures expressiveness of generated full-body gestures. Our framework also learns to denoise root noise conditioned on local body representation, guaranteeing stability and realism. DIDiffGes can generate gestures from speech with just 10 sampling steps, without compromising quality and expressiveness, reducing the number of sampling steps by a factor of 100 compared to existing methods. Our user study reveals that our method outperforms state-of-the-art approaches in human likeness, appropriateness, and style correctness. Project is https://cyk990422.github.io/DIDiffGes.

📄 PDF Abstract BibTeX arXiv:2503.17059

Code (0)

등록된 구현이 없습니다.

Tasks

Gesture Generation

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

$α$Surf: Implicit Surface Reconstruction for Semi-Transparent and Thin Objects with Decoupled Geometry and Opacity

2023-03-17 · Tianhao Wu, Hanxue Liang, Fangcheng Zhong, Gernot Riegler 외

Implicit surface representations such as the signed distance function (SDF) have emerged as a promising approach for image-based surface reconstruction. However, existing optimization methods assume solid surfaces and ar…

NeRF Surface Reconstruction

AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusion

2023-07-13 · Shuo Huang, Zongxin Yang, Liangting Li, Yi Yang 외

Large-scale pre-trained vision-language models allow for the zero-shot text-based generation of 3D avatars. The previous state-of-the-art method utilized CLIP to supervise neural implicit models that reconstructed a huma…

Decoupled Diffusion Sampling for Inverse Problems on Function Spaces

2026-01-30 · Thomas Y. L. Lin, Jiachen Yao, Lufang Chiang, Julius Berner 외 arxiv

We propose a data-efficient, physics-aware generative framework in function space for inverse PDE problems. Existing plug-and-play diffusion posterior samplers represent physics implicitly through joint coefficient-solut…

SDMTL: Semi-Decoupled Multi-grained Trajectory Learning for 3D human motion prediction

2020-10-11 · Xiaoli Liu, Jianqin Yin

Predicting future human motion is critical for intelligent robots to interact with humans in the real world, and human motion has the nature of multi-granularity. However, most of the existing work either implicitly mode…

Human motion prediction motion prediction

Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration

2023-10-26 · NeurIPS 2023 11 · Longlin Yu, Tianyu Xie, Yu Zhu, Tong Yang 외

Semi-implicit variational inference (SIVI) has been introduced to expand the analytical variational families by defining expressive semi-implicit distributions in a hierarchical manner. However, the single-layer architec…

Bayesian Inference Variational Inference