The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks
The Neural Tangent Kernel (NTK) viewpoint is widely employed to analyze the training dynamics of overparameterized Physics-Informed Neural Networks (PINNs). However, unlike the case of linear Partial Differential Equations (PDEs), we show how the NTK perspective falls short in the nonlinear scenario. Specifically, we establish that the NTK yields a random matrix at initialization that is not constant during training, contrary to conventional belief. Another significant difference from the linear regime is that, even in the idealistic infinite-width limit, the Hessian does not vanish and hence it cannot be disregarded during training. This motivates the adoption of second-order optimization methods. We explore the convergence guarantees of such methods in both linear and nonlinear cases, addressing challenges such as spectral bias and slow convergence. Every theoretical result is supported by numerical examples with both linear and nonlinear PDEs, and we highlight the benefits of second-order methods in benchmark test cases.
Code (0)
등록된 구현이 없습니다.
Tasks
Second-order methodsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Residual-loss Anomaly Analysis of Physics-Informed Neural Networks: An Inverse Method for Change-point Detection in Nonlinear Dynamical Systems with Regime Switching
Nonlinear dynamical systems with regime transitions are typically described by ordinary differential equations with jumping parameters parameters. Traditional methods often treat change-point detection and parameter esti…
Stiff Transfer Learning for Physics-Informed Neural Networks
Stiff differential equations are prevalent in various scientific domains, posing significant challenges due to the disparate time scales of their components. As computational power grows, physics-informed neural networks…
Computational EfficiencyTransfer LearningPhysics-informed neural networks for high-dimensional solutions and snaking bifurcations in nonlinear lattices
This paper introduces a framework based on physics-informed neural networks (PINNs) for addressing key challenges in nonlinear lattices, including solution approximation, bifurcation diagram construction, and linear stab…
Computational EfficiencyGRINN: A Physics-Informed Neural Network for solving hydrodynamic systems in the presence of self-gravity
Modeling self-gravitating gas flows is essential to answering many fundamental questions in astrophysics. This spans many topics including planet-forming disks, star-forming clouds, galaxy formation, and the development …
Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equations
Solving nonlinear partial differential equations (PDEs) with multiple solutions using neural networks has found widespread applications in various fields such as physics, biology, and engineering. However, classical neur…