Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation
We propose a novel unsupervised method to autoregressively generate videos from a single frame and a sparse motion input. Our trained model can generate unseen realistic object-to-object interactions. Although our model has never been given the explicit segmentation and motion of each object in the scene during training, it is able to implicitly separate their dynamics and extents. Key components in our method are the randomized conditioning scheme, the encoding of the input motion control, and the randomized and sparse sampling to enable generalization to out of distribution but realistic correlations. Our model, which we call YODA, has therefore the ability to move objects without physically touching them. Through extensive qualitative and quantitative evaluations on several datasets, we show that YODA is on par with or better than state of the art video generation prior work in terms of both controllability and video quality.
Code (1)
Tasks
ObjectVideo GenerationSimilar Papers 제목 키워드 기반
Training-free Controllable Human Motion Generation under Heterogeneous Constraints
Training-free controllable motion generation has attracted growing interest for enabling flexible constraint enforcement without constraint-specific training. However, existing training-free methods require constraints t…
M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking
Building a general-purpose whole-body controller is essential for enabling diverse motion capabilities in humanoid robots across a wide range of downstream tasks, including locomotion and loco-manipulation. Different tas…
Reinforcement LearningTask-space model-based control of pneumatic soft actuators
Soft actuators enable dexterous and compliant interaction, but closed-loop task-space control remains challenging due to strong nonlinearities, distributed deformation, and uncertainty in their dynamics. This paper prese…
Computational EfficiencyControllable Human-Object Interaction Synthesis
Synthesizing semantic-aware, long-horizon, human-object interaction is critical to simulate realistic human behaviors. In this work, we address the challenging problem of generating synchronized object motion and human m…
Human-Object Interaction DetectionObjectLearning Dexterous Grasping from Sparse Taxonomy Guidance
Dexterous manipulation requires planning a grasp configuration suited to the object and task, which is then executed through coordinated multi-finger control. However, specifying grasp plans with dense pose or contact ta…
Reinforcement Learning