End-to-End Human Pose and Mesh Reconstruction with Transformers
We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3D human pose and mesh vertices from a single image. Our method uses a transformer encoder to jointly model vertex-vertex and vertex-joint interactions, and outputs 3D joint coordinates and mesh vertices simultaneously. Compared to existing techniques that regress pose and shape parameters, METRO does not rely on any parametric mesh models like SMPL, thus it can be easily extended to other objects such as hands. We further relax the mesh topology and allow the transformer self-attention mechanism to freely attend between any two vertices, making it possible to learn non-local relationships among mesh vertices and joints. With the proposed masked vertex modeling, our method is more robust and effective in handling challenging situations like partial occlusions. METRO generates new state-of-the-art results for human mesh reconstruction on the public Human3.6M and 3DPW datasets. Moreover, we demonstrate the generalizability of METRO to 3D hand reconstruction in the wild, outperforming existing state-of-the-art methods on FreiHAND dataset. Code and pre-trained models are available at https://github.com/microsoft/MeshTransformer.
Code (1)
Tasks
3D Absolute Human Pose Estimation3D Hand Pose Estimation3D Human Pose EstimationSimilar Papers 제목 키워드 기반
A Lightweight Graph Transformer Network for Human Mesh Reconstruction from 2D Human Pose
Existing deep learning-based human mesh reconstruction approaches have a tendency to build larger networks in order to achieve higher accuracy. Computational complexity and model size are often neglected, despite being k…
3D Human Pose Estimation3D Human Shape EstimationHuman Mesh RecoveryVirtual Try-onA Modular Multi-stage Lightweight Graph Transformer Network for Human Pose and Shape Estimation from 2D Human Pose
In this research, we address the challenge faced by existing deep learning-based human mesh reconstruction methods in balancing accuracy and computational efficiency. These methods typically prioritize accuracy, resultin…
Computational EfficiencyMesh Graphormer
We present a graph-convolution-reinforced transformer, named Mesh Graphormer, for 3D human pose and mesh reconstruction from a single image. Recently both transformers and graph convolutional neural networks (GCNNs) have…
3D Hand Pose Estimation3D Human Pose EstimationMPT: Mesh Pre-Training with Transformers for Human Pose and Mesh Reconstruction
Traditional methods of reconstructing 3D human pose and mesh from single images rely on paired image-mesh datasets, which can be difficult and expensive to obtain. Due to this limitation, model scalability is constrained…
3D Human Pose EstimationHand Pose EstimationPose EstimationCross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformers
Transformer encoder architectures have recently achieved state-of-the-art results on monocular 3D human mesh reconstruction, but they require a substantial number of parameters and expensive computations. Due to the larg…
3D Hand Pose Estimation3D Human Pose Estimation3D ReconstructionDecoder