HandDiff: 3D Hand Pose Estimation with Diffusion on Image-Point Cloud

2024-04-04 · CVPR 2024 1 · Wencan Cheng, Hao Tang, Luc van Gool, Jong Hwan Ko

Extracting keypoint locations from input hand frames, known as 3D hand pose estimation, is a critical task in various human-computer interaction applications. Essentially, the 3D hand pose estimation can be regarded as a 3D point subset generative problem conditioned on input frames. Thanks to the recent significant progress on diffusion-based generative models, hand pose estimation can also benefit from the diffusion model to estimate keypoint locations with high quality. However, directly deploying the existing diffusion models to solve hand pose estimation is non-trivial, since they cannot achieve the complex permutation mapping and precise localization. Based on this motivation, this paper proposes HandDiff, a diffusion-based hand pose estimation model that iteratively denoises accurate hand pose conditioned on hand-shaped image-point clouds. In order to recover keypoint permutation and accurate location, we further introduce joint-wise condition and local detail condition. Experimental results demonstrate that the proposed HandDiff significantly outperforms the existing approaches on four challenging hand pose benchmark datasets. Codes and pre-trained models are publicly available at https://github.com/cwc1260/HandDiff.

📄 PDF Abstract BibTeX arXiv:2404.03159

Code (1)

cwc1260/handdiff 공식 구현 pytorch

Tasks

3D Hand Pose Estimation Hand Pose Estimation Pose Estimation

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models

2023-12-08 · Pei Lin, Sihang Xu, Hongdi Yang, Yiran Liu 외

Existing hands datasets are largely short-range and the interaction is weak due to the self-occlusion and self-similarity of hands, which can not yet fit the need for interacting hands motion generation. To rescue the da…

Data Augmentation Motion Generation motion in-betweening Temporal Sequences

DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation

2024-11-27 · CVPR 2025 1 · Zhixuan Liang, Yao Mu, Yixiao Wang, Tianxing Chen 외

Dexterous manipulation with contact-rich interactions is crucial for advanced robotics. While recent diffusion-based planning approaches show promise for simple manipulation tasks, they often produce unrealistic ghost st…

Contact-rich Manipulation

Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D

2023-12-02 · Karran Pandey, Paul Guerrero, Matheus Gadelha, Yannick Hold-Geoffroy 외

Diffusion Handles is a novel approach to enabling 3D object edits on diffusion images. We accomplish these edits using existing pre-trained diffusion models, and 2D image depth estimation, without any fine-tuning or 3D o…

3D Object Retrieval Depth Estimation Object Retrieval

Denoising Diffusion for 3D Hand Pose Estimation from Images

2023-08-18 · Maksym Ivashechkin, Oscar Mendez, Richard Bowden

Hand pose estimation from a single image has many applications. However, approaches to full 3D body pose estimation are typically trained on day-to-day activities or actions. As such, detailed hand-to-hand interactions a…

3D Hand Pose Estimation Denoising Hand Pose Estimation Image to 3D+1

Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D

2024-01-01 · CVPR 2024 1 · Karran Pandey, Paul Guerrero, Matheus Gadelha, Yannick Hold-Geoffroy 외

Diffusion handles is a novel approach to enable 3D object edits on diffusion images requiring only existing pre-trained diffusion models depth estimation without any fine-tuning or 3D object retrieval. The edited res…