Video-based Human-Object Interaction Detection from Tubelet Tokens

2022-06-04 · Danyang Tu, Wei Sun, Xiongkuo Min, Guangtao Zhai, Wei Shen

We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatiotemporal representations, for video-based human-object interaction (V-HOI) detection. The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens along spatial and temporal domains, which enjoy two benefits: 1) Compactness: each tubelet token is learned by a selective attention mechanism to reduce redundant spatial dependencies from others; 2) Expressiveness: each tubelet token is enabled to align with a semantic instance, i.e., an object or a human, across frames, thanks to agglomeration and linking. The effectiveness and efficiency of TUTOR are verified by extensive experiments. Results shows our method outperforms existing works by large margins, with a relative mAP gain of $16.14\%$ on VidHOI and a 2 points gain on CAD-120 as well as a $4 \times$ speedup.

📄 PDF Abstract BibTeX arXiv:2206.01908

Code (0)

등록된 구현이 없습니다.

Tasks

Human-Object Interaction Detection

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Adam 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Social Fabric: Tubelet Compositions for Video Relation Detection

2021-08-18 · ICCV 2021 10 · Shuo Chen, Zenglin Shi, Pascal Mettes, Cees G. M. Snoek

This paper strives to classify and detect the relationship between object tubelets appearing within a video as a <subject-predicate-object> triplet. Where existing works treat object proposals or tubelets as single entit…

Object Relation Triplet Video Visual Relation Detection+1

Object Detection in Videos with Tubelet Proposal Networks

2017-02-21 · CVPR 2017 7 · Kai Kang, Hongsheng Li, Tong Xiao, Wanli Ouyang 외

Object detection in videos has drawn increasing attention recently with the introduction of the large-scale ImageNet VID dataset. Different from object detection in static images, temporal information in videos is vital …

Object object-detection Object Detection Object Tracking

Object Detection in Videos by High Quality Object Linking

2018-01-30 · Peng Tang, Chunyu Wang, Xinggang Wang, Wenyu Liu 외

Compared with object detection in static images, object detection in videos is more challenging due to degraded image qualities. An effective way to address this problem is to exploit temporal contexts by linking the sam…

General Classification Object object-detection Object Detection+1

Object Detection in Videos by Short and Long Range Object Linking

2018-01-30 · IEEE Transactions on Pattern Analysis and Machine Intelligence(TPAM) 2018 1 · Peng Tang † Chunyu Wang ‡ Xinggang Wang † Wenyu Liu † Wenjun Zeng ‡ Jingdong Wang ‡ † School of EIC, Huazhong University of Science and Technology ‡ Microsoft Research Asia

We address the problem of detecting objects in videos with the interest in exploring temporal contexts. Our core idea is to link objects in the short and long ranges for improving the classification quality. Our approach…

Classification Object object-detection Object Detection

In Defense of Clip-based Video Relation Detection

2023-07-18 · Meng Wei, Long Chen, Wei Ji, Xiaoyu Yue 외

Video Visual Relation Detection (VidVRD) aims to detect visual relationship triplets in videos using spatial bounding boxes and temporal boundaries. Existing VidVRD methods can be broadly categorized into bottom-up and t…

Feature Compression Object Tracking Relation Video Visual Relation Detection