Patch-based Object-centric Transformers for Efficient Video Generation
In this work, we present Patch-based Object-centric Video Transformer (POVT), a novel region-based video generation architecture that leverages object-centric information to efficiently model temporal dynamics in videos. We build upon prior work in video prediction via an autoregressive transformer over the discrete latent space of compressed videos, with an added modification to model object-centric information via bounding boxes. Due to better compressibility of object-centric representations, we can improve training efficiency by allowing the model to only access object information for longer horizon temporal information. When evaluated on various difficult object-centric datasets, our method achieves better or equal performance to other video generation models, while remaining computationally more efficient and scalable. In addition, we show that our method is able to perform object-centric controllability through bounding box manipulation, which may aid downstream tasks such as video editing, or visual planning. Samples are available at https://sites.google.com/view/povt-public
Code (1)
Tasks
ObjectVideo EditingVideo GenerationVideo PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Object-Region Video Transformers
Recently, video transformers have shown great success in video understanding, exceeding CNN performance; yet existing video transformer models do not explicitly model objects, although objects can be essential for recogn…
Action DetectionAction RecognitionFew-Shot action recognitionFew Shot Action Recognition+2DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particles
We propose a new object-centric video prediction algorithm based on the deep latent particle (DLP) representation. In comparison to existing slot- or patch-based representations, DLPs model the scene using a set of keypo…
ObjectPositionUnconditional Video GenerationVideo Generation+1Finding Distributed Object-Centric Properties in Self-Supervised Transformers
Self-supervised Vision Transformers (ViTs) like DINO show an emergent ability to discover objects, typically observed in [CLS] token attention maps of the final layer. However, these maps often contain spurious activatio…
Visual GroundingLearning Explicit Object-Centric Representations with Vision Transformers
With the recent successful adaptation of transformers to the vision domain, particularly when trained in a self-supervised fashion, it has been shown that vision transformers can learn impressive object-reasoning-like be…
ObjectSegmentationSemantic SegmentationGenerative Video Transformer: Can Objects be the Words?
Transformers have been successful for many natural language processing tasks. However, applying transformers to the video domain for tasks such as long-term video generation and scene understanding has remained elusive d…
GPUScene UnderstandingVideo Generation