HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects
We construct the first markerless deformable interaction dataset recording interactive motions of the hands and deformable objects, called HMDO (Hand Manipulation with Deformable Objects). With our built multi-view capture system, it captures the deformable interactions with multiple perspectives, various object shapes, and diverse interactive forms. Our motivation is the current lack of hand and deformable object interaction datasets, as 3D hand and deformable object reconstruction is challenging. Mainly due to mutual occlusion, the interaction area is difficult to observe, the visual features between the hand and the object are entangled, and the reconstruction of the interaction area deformation is difficult. To tackle this challenge, we propose a method to annotate our captured data. Our key idea is to collaborate with estimated hand features to guide the object global pose estimation, and then optimize the deformation process of the object by analyzing the relationship between the hand and the object. Through comprehensive evaluation, the proposed method can reconstruct interactive motions of hands and deformable objects with high quality. HMDO currently consists of 21600 frames over 12 sequences. In the future, this dataset could boost the research of learning-based reconstruction of deformable interaction scenes.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectObject ReconstructionPose EstimationSimilar Papers 제목 키워드 기반
Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements
Markerless motion capture (MMC) techniques have been widely beneficial in biomechanical analysis of human movement; however, application to complex motions of the hand lags other musculoskeletal systems. The primary goal…
Pose EstimationCapture Dense: Markerless Motion Capture Meets Dense Pose Estimation
We present a method to combine markerless motion capture and dense pose feature estimation into a single framework. We demonstrate that dense pose information can help for multiview/single-view motion capture, and multiv…
Human ParsingMarkerless Motion CapturePose EstimationHOH: Markerless Multimodal Human-Object-Human Handover Dataset with Large Object Count
We present the HOH (Human-Object-Human) Handover Dataset, a large object count dataset with 136 objects, to accelerate data-driven research on handover studies, human-robot handover implementation, and artificial intelli…
Objectparameter estimationTrajectory PredictionCtRNet-X: Camera-to-Robot Pose Estimation in Real-world Conditions Using a Single Camera
Camera-to-robot calibration is crucial for vision-based robot control and requires effort to make it accurate. Recent advancements in markerless pose estimation methods have eliminated the need for time-consuming physica…
Pose EstimationRobot Pose EstimationMANUS: Markerless Grasp Capture using Articulated 3D Gaussians
Understanding how we grasp objects with our hands has important applications in areas like robotics and mixed reality. However, this challenging problem requires accurate modeling of the contact between hands and objects…
Mixed RealityObject