Mumpy: Multilateral Temporal-view Pyramid Transformer for Video Inpainting Detection
The task of video inpainting detection is to expose the pixel-level inpainted regions within a video sequence. Existing methods usually focus on leveraging spatial and temporal inconsistencies. However, these methods typically employ fixed operations to combine spatial and temporal clues, limiting their applicability in different scenarios. In this paper, we introduce a novel Multilateral Temporal-view Pyramid Transformer ({\em MumPy}) that collaborates spatial-temporal clues flexibly. Our method utilizes a newly designed multilateral temporal-view encoder to extract various collaborations of spatial-temporal clues and introduces a deformable window-based temporal-view interaction module to enhance the diversity of these collaborations. Subsequently, we develop a multi-pyramid decoder to aggregate the various types of features and generate detection maps. By adjusting the contribution strength of spatial and temporal clues, our method can effectively identify inpainted regions. We validate our method on existing datasets and also introduce a new challenging and large-scale Video Inpainting dataset based on the YouTube-VOS dataset, which employs several more recent inpainting methods. The results demonstrate the superiority of our method in both in-domain and cross-domain evaluation scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderDiversityVideo InpaintingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Driving
Vision-centric joint perception and prediction (PnP) has become an emerging trend in autonomous driving research. It predicts the future states of the traffic participants in the surrounding environment from raw RGB imag…
Autonomous DrivingBird's-Eye View Semantic SegmentationVisualizing the Passage of Time with Video Temporal Pyramids
What can we learn about a scene by watching it for months or years? A video recorded over a long timespan will depict interesting phenomena at multiple timescales, but identifying and viewing them presents a challenge. T…
Refined Temporal Pyramidal Compression-and-Amplification Transformer for 3D Human Pose Estimation
Accurately estimating the 3D pose of humans in video sequences requires both accuracy and a well-structured architecture. With the success of transformers, we introduce the Refined Temporal Pyramidal Compression-and-Ampl…
3D Human Pose EstimationPose EstimationPRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting
The self-attention mechanism in Transformer architecture, invariant to sequence order, necessitates positional embeddings to encode temporal order in time series prediction. We argue that this reliance on positional embe…
Multivariate Time Series ForecastingTemporal SequencesTime SeriesTime Series Forecasting+1Efficient End-to-End Video Question Answering with Pyramidal Multimodal Transformer
This paper presents a new method for end-to-end Video Question Answering (VideoQA), aside from the current popularity of using large-scale pre-training with huge feature extractors. We achieve this with a pyramidal multi…
Computational EfficiencyQuestion AnsweringRetrievalText to Video Retrieval+2