$Δ$ynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos
Inferring rigid-body physical states and properties from monocular videos is a fundamental step toward physics-based perception and simulation. Existing approaches assume specific underlying physical systems, object types, and camera poses, making them unable to generalize to complex real-world settings. We introduce $Δ$YNAMICS, a vision-language framework that uses language as a unified representation of rigid-body dynamics. Instead of directly predicting parameters, $Δ$YNAMICS generates scene configurations in a structured text format for physics simulation. We enhance the model's generalization by integrating natural language motion reasoning and leveraging optical flow as a semantic-agnostic input. On the CLEVRER dataset, $Δ$YNAMICS achieves a segmentation IoU of 0.30, a 7x improvement over leading VLMs (InternVL3-8B, Qwen2.5-VL-7B and Claude-4-Sonnet). Additionally, test-time sampling and evolutionary search further boost performance by 27% and 120% in segmentation IoU, respectively. Finally, we demonstrate strong transfer to a new dataset of 235 real-world rigid-body videos, highlighting the potential of language-driven physics inference for bridging perception and simulation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
RigidFormer: Learning Rigid Dynamics using Transformers
Learning-based simulation of multi-object rigid-body dynamics remains difficult because contact is discontinuous and errors compound over long horizons. Most existing methods remain tied to mesh connectivity and vertex-l…
Point CloudsPredicting Rigid Body Dynamics using Dual Quaternion Recurrent Neural Networks with Quaternion Attention
We propose a novel neural network architecture based on dual quaternions which allow for a compact representation of informations with a main focus on describing rigid body movements. To cover the dynamic behavior inhere…
PositionKoopman Operator Based Modeling and Control of Rigid Body Motion Represented by Dual Quaternions
In this paper, we systematically derive a finite set of Koopman based observables to construct a lifted linear state space model that describes the rigid body dynamics based on the dual quaternion representation. In gene…
Learning to Predict 3D Rotational Dynamics from Images of a Rigid Body with Unknown Mass Distribution
In many real-world settings, image observations of freely rotating 3D rigid bodies may be available when low-dimensional measurements are not. However, the high-dimensionality of image data precludes the use of classical…
Pseudo-rigid body networks: learning interpretable deformable object dynamics from partial observations
Accurately predicting deformable linear object (DLO) dynamics is challenging, especially when the task requires a model that is both human-interpretable and computationally efficient. In this work, we draw inspiration fr…
Decoder