Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space
Learning causal relationships in high-dimensional data (images, videos) is a hard task, as they are often defined on low dimensional manifolds and must be extracted from complex signals dominated by appearance, lighting, textures and also spurious correlations in the data. We present a method for learning counterfactual reasoning of physical processes in pixel space, which requires the prediction of the impact of interventions on initial conditions. Going beyond the identification of structural relationships, we deal with the challenging problem of forecasting raw video over long horizons. Our method does not require the knowledge or supervision of any ground truth positions or other object or scene properties. Our model learns and acts on a suitable hybrid latent representation based on a combination of dense features, sets of 2D keypoints and an additional latent vector per keypoint. We show that this better captures the dynamics of physical processes than purely dense or sparse representations. We introduce a new challenging and carefully designed counterfactual benchmark for predictions in pixel space and outperform strong baselines in physics-inspired ML and video prediction.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualCounterfactual ReasoningVideo PredictionSimilar Papers 제목 키워드 기반
CoPhy: Counterfactual Learning of Physical Dynamics
Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world. This work poses a new problem of counterfactual learning of object mechanics from visual input. We deve…
counterfactualVideo PredictionCausal Counterfactuals for Improving the Robustness of Reinforcement Learning
Reinforcement learning (RL) is used in various robotic applications. RL enables agents to learn tasks autonomously by interacting with the environment. The more critical the tasks are, the higher the demand for the robus…
Causal Inferencereinforcement-learningReinforcement LearningReinforcement Learning (RL)+1CoPhy-PGNN: Learning Physics-guided Neural Networks with Competing Loss Functions for Solving Eigenvalue Problems
Physics-guided Neural Networks (PGNNs) represent an emerging class of neural networks that are trained using physics-guided (PG) loss functions (capturing violations in network outputs with known physics), along with the…
Physics-Inspired Unsupervised Classification for Region of Interest in X-Ray Ptychography
X-ray ptychography allows for large fields to be imaged at high resolution at the cost of additional computational expense due to the large volume of data. Given limited information regarding the object, the acquired dat…
ObjectCoFi-PGMA: Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs
Large language model (LLM) deployments increasingly rely on multi-agent architectures in which multiple models either compete through routing mechanisms or collaborate to produce a final answer. In both settings, the lea…