Total capture: 3D human pose estimation fusing video and inertial sensors
We present an algorithm for fusing multi-viewpoint video (MVV) with inertial measurement unit (IMU) sensor data to accurately estimate 3D human pose. A 3-D convolutional neural network is used to learn a pose embedding from volumetric probabilistic visual hull data (PVH) derived from the MVV frames. We incorporate this model within a dual stream network integrating pose embeddings derived from MVV and a forward kinematic solve of the IMU data. A temporal model (LSTM) is incorporated within both streams prior to their fusion. Hybrid pose inference using these two complementary data sources is shown to resolve ambiguities within each sensor modality, yielding improved accuracy over prior methods. A further contribution of this work is a new hybrid MVV dataset (TotalCapture) comprising video, IMU and a skeletal joint ground truth derived from a commercial motion capture system. The dataset is available online at http://cvssp.org/data/totalcapture/
Code (0)
등록된 구현이 없습니다.
Tasks
3D Human Pose EstimationPose EstimationSimilar Papers 제목 키워드 기반
Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation
This paper proposes a unified framework dubbed Multi-view and Temporal Fusing Transformer (MTF-Transformer) to adaptively handle varying view numbers and video length without camera calibration in 3D Human Pose Estimatio…
3D Human Pose EstimationCamera CalibrationPose EstimationDeepFuse: An IMU-Aware Network for Real-Time 3D Human Pose Estimation from Multi-View Image
In this paper, we propose a two-stage fully 3D network, namely \textbf{DeepFuse}, to estimate human pose in 3D space by fusing body-worn Inertial Measurement Unit (IMU) data and multi-view images deeply. The first stage …
3D Human Pose Estimation3D Pose EstimationPose EstimationMotionPRO: Exploring the Role of Pressure in Human MoCap and Beyond
Existing human Motion Capture (MoCap) methods mostly focus on the visual similarity while neglecting the physical plausibility. As a result, downstream tasks such as driving virtual human in 3D scene or humanoid robo…
Swin-Pose: Swin Transformer Based Human Pose Estimation
Convolutional neural networks (CNNs) have been widely utilized in many computer vision tasks. However, CNNs have a fixed reception field and lack the ability of long-range perception, which is crucial to human pose estim…
Pose EstimationAnti-Confusing: Region-Aware Network for Human Pose Estimation
In this work, we propose a novel framework named Region-Aware Network (RANet), which learns the ability of anti-confusing in case of heavy occlusion, nearby person and symmetric appearance, for human pose estimation. Spe…
Data AugmentationPose Estimation