DeVOS: Flow-Guided Deformable Transformer for Video Object Segmentation
The recent works on Video Object Segmentation achieved remarkable results by matching dense semantic and instance-level features between the current and previous frames for long-time propagation. Nevertheless, global feature matching ignores scene motion context, failing to satisfy temporal consistency. Even though some methods introduce local matching branch to achieve smooth propagation, they fail to model complex appearance changes due to the constraints of the local window. In this paper, we present DeVOS (Deformable VOS), an architecture for Video Object Segmentation that combines memory-based matching with motion-guided propagation resulting in stable long-term modeling and strong temporal consistency. For short-term local propagation, we propose a novel attention mechanism ADVA (Adaptive Deformable Video Attention), allowing the adaption of similarity search region to query-specific semantic features, which ensures robust tracking of complex shape and scale changes. DeVOS employs an optical flow to obtain scene motion features which are further injected to deformable attention as strong priors to learnable offsets. Our method achieves top-rank performance on DAVIS 2017 val and test-dev (88.1%, 83.0%), YouTube-VOS 2019 val (86.6%) while featuring consistent run-time speed and stable memory consumption
Code (0)
등록된 구현이 없습니다.
Tasks
Optical Flow EstimationSemantic SegmentationVideo Object SegmentationVideo Semantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FDAN: Flow-guided Deformable Alignment Network for Video Super-Resolution
Most Video Super-Resolution (VSR) methods enhance a video reference frame by aligning its neighboring frames and mining information on these frames. Recently, deformable alignment has drawn extensive attention in VSR com…
Optical Flow EstimationSuper-ResolutionVideo Super-ResolutionExploiting Optical Flow Guidance for Transformer-Based Video Inpainting
Transformers have been widely used for video processing owing to the multi-head self attention (MHSA) mechanism. However, the MHSA mechanism encounters an intrinsic difficulty for video inpainting, since the features ass…
Optical Flow EstimationVideo InpaintingBSRT: Improving Burst Super-Resolution with Swin Transformer and Flow-Guided Deformable Alignment
This work addresses the Burst Super-Resolution (BurstSR) task using a new architecture, which requires restoring a high-quality image from a sequence of noisy, misaligned, and low-resolution RAW bursts. To overcome the c…
Burst Image ReconstructionBurst Image Super-ResolutionMulti-Frame Super-ResolutionOptical Flow Estimation+1End-to-End Video Object Detection with Spatial-Temporal Transformers
Recently, DETR and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, …
Objectobject-detectionObject DetectionOptical Flow Estimation+1Hybrid Local-Global Context Learning for Neural Video Compression
In neural video codecs, current state-of-the-art methods typically adopt multi-scale motion compensation to handle diverse motions. These methods estimate and compress either optical flow or deformable offsets to reduce …
Motion CompensationMotion EstimationOptical Flow EstimationVideo Compression