TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation
Text-driven Image to Video Generation (TI2V) aims to generate controllable video given the first frame and corresponding textual description. The primary challenges of this task lie in two parts: (i) how to identify the target objects and ensure the consistency between the movement trajectory and the textual description. (ii) how to improve the subjective quality of generated videos. To tackle the above challenges, we propose a new diffusion-based TI2V framework, termed TIV-Diffusion, via object-centric textual-visual alignment, intending to achieve precise control and high-quality video generation based on textual-described motion for different objects. Concretely, we enable our TIV-Diffuion model to perceive the textual-described objects and their motion trajectory by incorporating the fused textual and visual knowledge through scale-offset modulation. Moreover, to mitigate the problems of object disappearance and misaligned objects and motion, we introduce an object-centric textual-visual alignment module, which reduces the risk of misaligned objects/motion by decoupling the objects in the reference image and aligning textual features with each object individually. Based on the above innovations, our TIV-Diffusion achieves state-of-the-art high-quality video generation compared with existing TI2V methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Image to Video GenerationObjectVideo GenerationSimilar Papers 제목 키워드 기반
Intention-driven Ego-to-Exo Video Generation
Ego-to-exo video generation refers to generating the corresponding exocentric video according to the egocentric video, providing valuable applications in AR/VR and embodied AI. Benefiting from advancements in diffusion m…
Optical Flow EstimationStereo MatchingVideo GenerationHOIAnimator: Generating Text-prompt Human-object Animations using Novel Perceptive Diffusion Models
To date the quest to rapidly and effectively produce human-object interaction (HOI) animations directly from textual descriptions stands at the forefront of computer vision research. The underlying challenge demands …
DenoisingHuman-Object Interaction DetectionText-driven Affordance Learning from Egocentric Vision
Visual affordance learning is a key component for robots to understand how to interact with objects. Conventional approaches in this field rely on pre-defined objects and actions, falling short of capturing diverse inter…
Referring ExpressionReferring Expression ComprehensionVASE: Object-Centric Appearance and Shape Manipulation of Real Videos
Recently, several works tackled the video editing task fostered by the success of large-scale text-to-image generative models. However, most of these methods holistically edit the frame using the text, exploiting the pri…
Video EditingRating Continuous Actions in Spatial Multi-Agent Problems
We study credit assignment problems in spatial multi-agent environments where agents pursue a joint objective. On the example of soccer, we rate the movements of individual players with respect to their potential for sta…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)