Why Cannot Neural Networks Master Extrapolation? Insights from Physical Laws
Motivated by the remarkable success of Foundation Models (FMs) in language modeling, there has been growing interest in developing FMs for time series prediction, given the transformative power such models hold for science and engineering. This culminated in significant success of FMs in short-range forecasting settings. However, extrapolation or long-range forecasting remains elusive for FMs, which struggle to outperform even simple baselines. This contrasts with physical laws which have strong extrapolation properties, and raises the question of the fundamental difference between the structure of neural networks and physical laws. In this work, we identify and formalize a fundamental property characterizing the ability of statistical learning models to predict more accurately outside of their training domain, hence explaining performance deterioration for deep learning models in extrapolation settings. In addition to a theoretical analysis, we present empirical results showcasing the implications of this property on current deep learning architectures. Our results not only clarify the root causes of the extrapolation gap but also suggest directions for designing next-generation forecasting models capable of mastering extrapolation.
Code (0)
등록된 구현이 없습니다.
Tasks
Time Series PredictionSimilar Papers 제목 키워드 기반
DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration
Learned world models excel at interpolative generalization but fail at extrapolative generalization to novel physical properties. This limitation arises because they learn statistical correlations rather than the environ…
PhysMaster: Mastering Physical Representation for Video Generation via Reinforcement Learning
Video generation models nowadays are capable of generating visually realistic videos, but often fail to adhere to physical laws, limiting their ability to generate physically plausible videos and serve as ''world models'…
Representation LearningReinforcement LearningVideo GenerationExcluding the Target Domain Improves Extrapolation: Deconfounded Hierarchical Physics Constraints
Extrapolation to out-of-distribution conditions is a fundamental challenge for physics-constrained deep generative models. Existing methods apply physical constraints as a single static regularization term uniformly acro…
Modular Lagrangian Neural Networks: Designing Structures of Networks with Physical Inductive Biases
Deep learning struggles at extrapolation in many cases. This issue happens when it comes to untrained data domains or different input dimensions and becomes more common in physical problems. Leveraging physical inductive…
Improving physics-informed neural network extrapolation via transfer learning and adaptive activation functions
Physics-Informed Neural Networks (PINNs) are deep learning models that incorporate the governing physical laws of a system into the learning process, making them well-suited for solving complex scientific and engineering…
Transfer Learning