Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation

2023-03-22 · ICCV 2023 1 · Xiangtai Li, Haobo Yuan, Wenwei Zhang, Guangliang Cheng, Jiangmiao Pang, Chen Change Loy

Video segmentation aims to segment and track every pixel in diverse scenarios accurately. In this paper, we present Tube-Link, a versatile framework that addresses multiple core tasks of video segmentation with a unified architecture. Our framework is a near-online approach that takes a short subclip as input and outputs the corresponding spatial-temporal tube masks. To enhance the modeling of cross-tube relationships, we propose an effective way to perform tube-level linking via attention along the queries. In addition, we introduce temporal contrastive learning to instance-wise discriminative features for tube-level association. Our approach offers flexibility and efficiency for both short and long video inputs, as the length of each subclip can be varied according to the needs of datasets or scenarios. Tube-Link outperforms existing specialized architectures by a significant margin on five video segmentation datasets. Specifically, it achieves almost 13% relative improvements on VIPSeg and 4% improvements on KITTI-STEP over the strong baseline Video K-Net. When using a ResNet50 backbone on Youtube-VIS-2019 and 2021, Tube-Link boosts IDOL by 3% and 4%, respectively.

📄 PDF Abstract BibTeX arXiv:2303.12782

Code (1)

lxtgh/tube-link 공식 구현 pytorch

Tasks

Contrastive Learning Segmentation Video Instance Segmentation Video Panoptic Segmentation Video Segmentation Video Semantic Segmentation

Methods 이 논문이 사용한 방법론

Contrastive Learning 설명 없음

K-Net K-Net is a framework for unified semantic and instance segmentation that segments both instances and semantic categories consistently by a group of learnable kernels, where…

Deformable Tube Network for Action Detection in Videos

2019-07-03 · Wei Li, Zehuan Yuan, Dashan Guo, Lei Huang 외

We address the problem of spatio-temporal action detection in videos. Existing methods commonly either ignore temporal context in action recognition and localization, or lack the modelling of flexible shapes of action tu…

Action Detection Action Recognition

Object Detection in Videos by High Quality Object Linking

2018-01-30 · Peng Tang, Chunyu Wang, Xinggang Wang, Wenyu Liu 외

Compared with object detection in static images, object detection in videos is more challenging due to degraded image qualities. An effective way to address this problem is to exploit temporal contexts by linking the sam…

General Classification Object object-detection Object Detection+1

Object Detection in Videos by Short and Long Range Object Linking

2018-01-30 · IEEE Transactions on Pattern Analysis and Machine Intelligence(TPAM) 2018 1 · Peng Tang † Chunyu Wang ‡ Xinggang Wang † Wenyu Liu † Wenjun Zeng ‡ Jingdong Wang ‡ † School of EIC, Huazhong University of Science and Technology ‡ Microsoft Research Asia

We address the problem of detecting objects in videos with the interest in exploring temporal contexts. Our core idea is to link objects in the short and long ranges for improving the classification quality. Our approach…

Classification Object object-detection Object Detection

Video-based Human-Object Interaction Detection from Tubelet Tokens

2022-06-04 · Danyang Tu, Wei Sun, Xiongkuo Min, Guangtao Zhai 외

We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatiotemporal representations, for video-based human-object interaction (V-HOI) detection. The tubel…

Human-Object Interaction Detection

Predicting Action Tubes

2018-08-23 · Gurkirt Singh, Suman Saha, Fabio Cuzzolin

In this work, we present a method to predict an entire `action tube' (a set of temporally linked bounding boxes) in a trimmed video just by observing a smaller subset of it. Predicting where an action is going to take pl…

Action Classification Action Detection Autonomous Driving