End-to-End Trainable Multi-Instance Pose Estimation with Transformers

2021-03-22 · Lucas Stoffl, Maxime Vidal, Alexander Mathis

We propose a new end-to-end trainable approach for multi-instance pose estimation by combining a convolutional neural network with a transformer. We cast multi-instance pose estimation from images as a direct set prediction problem. Inspired by recent work on end-to-end trainable object detection with transformers, we use a transformer encoder-decoder architecture together with a bipartite matching scheme to directly regress the pose of all individuals in a given image. Our model, called POse Estimation Transformer (POET), is trained using a novel set-based global loss that consists of a keypoint loss, a keypoint visibility loss, a center loss and a class loss. POET reasons about the relations between detected humans and the full image context to directly predict the poses in parallel. We show that POET can achieve high accuracy on the challenging COCO keypoint detection task. To the best of our knowledge, this model is the first end-to-end trainable multi-instance human pose estimation method.

📄 PDF Abstract BibTeX arXiv:2103.12115

Code (2)

amathislab/poet 공식 구현 pytorch

pranoyr/pose-estimation-with-transformers pytorch

Tasks

Decoder Keypoint Detection object-detection Object Detection Pose Estimation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Residual Connection 설명 없음

Adam 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

COPE: End-to-end trainable Constant Runtime Object Pose Estimation

2022-08-18 · Stefan Thalhammer, Timothy Patten, Markus Vincze

State-of-the-art object pose estimation handles multiple instances in a test image by using multi-model formulations: detection as a first stage and then separately trained networks per object for 2D-3D geometric corresp…

6D Pose Estimation using RGB Object Pose Estimation

Learning Spectral Methods by Transformers

2025-01-02 · Yihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu 외

Transformers demonstrate significant advantages as the building block of modern LLMs. In this work, we study the capacities of Transformers in performing unsupervised learning. We show that multi-layered Transformers, gi…

In-Context Learning

An End-to-End Trainable Video Panoptic Segmentation Method usingTransformers

2021-10-08 · Jeongwon Ryu, Kwangjin Yoon

In this paper, we present an algorithm to tackle a video panoptic segmentation problem, a newly emerging area of research. The video panoptic segmentation is a task that unifies the typical task of panoptic segmentation …

Multi-Object Tracking Object Tracking Panoptic Segmentation Segmentation+1

InsPose: Instance-Aware Networks for Single-Stage Multi-Person Pose Estimation

2021-07-19 · Dahu Shi, Xing Wei, Xiaodong Yu, Wenming Tan 외

Multi-person pose estimation is an attractive and challenging task. Existing methods are mostly based on two-stage frameworks, which include top-down and bottom-up methods. Two-stage methods either suffer from high compu…

Multi-Person Pose Estimation Pose Estimation

Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry

2024-09-13 · Yunus Bilge Kurt, Ahmet Akman, A. Aydin Alatan

In recent years, transformer-based architectures become the de facto standard for sequence modeling in deep learning frameworks. Inspired by the successful examples, we propose a causal visual-inertial fusion transformer…

Pose Estimation