CamViG: Camera Aware Image-to-Video Generation with Multimodal Transformers
We extend multimodal transformers to include 3D camera motion as a conditioning signal for the task of video generation. Generative video models are becoming increasingly powerful, thus focusing research efforts on methods of controlling the output of such models. We propose to add virtual 3D camera controls to generative video methods by conditioning generated video on an encoding of three-dimensional camera movement over the course of the generated video. Results demonstrate that we are (1) able to successfully control the camera during video generation, starting from a single frame and a camera signal, and (2) we demonstrate the accuracy of the generated 3D camera paths using traditional computer vision methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Image to Video GenerationVideo GenerationSimilar Papers 제목 키워드 기반
CineScene: Implicit 3D as Effective Scene Representation for Cinematic Video Generation
Cinematic video production requires control over scene-subject composition and camera movement, but live-action shooting remains costly due to the need for constructing physical sets. To address this, we introduce the ta…
Text-to-Video GenerationCamContextI2V: Context-aware Controllable Video Generation
Recently, image-to-video (I2V) diffusion models have demonstrated impressive scene understanding and generative quality, incorporating image conditions to guide generation. However, these models primarily animate static …
DiversityScene UnderstandingVideo GenerationMotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation
This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movemen…
Image to Video GenerationVideo EditingVideo GenerationJOG3R: Towards 3D-Consistent Video Generators
Emergent capabilities of image generators have led to many impactful zero- or few-shot applications. Inspired by this success, we investigate whether video generators similarly exhibit 3D-awareness. Using structure-from-…
Camera Pose EstimationPose EstimationVideo GenerationCRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation
Camera-conditioned video generation requires positional encoding that remains reliable under changes in camera motion, lens configuration, and scene structure. However, existing attention-level camera encodings either pr…
Video Generation