KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images
We present KeyCLD, a framework to learn Lagrangian dynamics from images. Learned keypoints represent semantic landmarks in images and can directly represent state dynamics. We show that interpreting this state as Cartesian coordinates, coupled with explicit holonomic constraints, allows expressing the dynamics with a constrained Lagrangian. KeyCLD is trained unsupervised end-to-end on sequences of images. Our method explicitly models the mass matrix, potential energy and the input matrix, thus allowing energy based control. We demonstrate learning of Lagrangian dynamics from images on the dm_control pendulum, cartpole and acrobot environments. KeyCLD can be learned on these systems, whether they are unactuated, underactuated or fully actuated. Trained models are able to produce long-term video predictions, showing that the dynamics are accurately learned. We compare with Lag-VAE, Lag-caVAE and HGN, and investigate the benefit of the Lagrangian prior and the constraint function. KeyCLD achieves the highest valid prediction time on all benchmarks. Additionally, a very straightforward energy shaping controller is successfully applied on the fully actuated systems. Please refer to our project page for code and additional results: https://rdaems.github.io/keycld/
Code (0)
등록된 구현이 없습니다.
Tasks
AcrobotvalidSimilar Papers 제목 키워드 기반
Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control
Recent approaches for modelling dynamics of physical systems with neural networks enforce Lagrangian or Hamiltonian structure to improve prediction and generalization. However, when coordinates are embedded in high-dimen…
PredictionFrom Fixed Grids to Moving Particles:A Transferable Latent Operator for Fluid Dynamics
Lagrangian modeling is vital to fluid dynamics, as it characterizes particle transport and complements the Eulerian representation. However, Lagrangian trajectories are less commonly available than Eulerian fields, while…
Robust Lagrangian and Adversarial Policy Gradient for Robust Constrained Markov Decision Processes
The robust constrained Markov decision process (RCMDP) is a recent task-modelling framework for reinforcement learning that incorporates behavioural constraints and that provides robustness to errors in the transition dy…
Incremental LearningSimplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints
Reasoning about the physical world requires models that are endowed with the right inductive biases to learn the underlying dynamics. Recent works improve generalization for predicting trajectories by learning the Hamilt…
LaWM: Least Action World Models for Long-Horizon Physical Consistency from Visual Observations
Learning predictive world models from visual observations is a core problem in embodied AI, with applications to model-based reinforcement learning and robotic planning. Existing latent world models typically generate fu…
Reinforcement LearningVideo Generation