The Differential Neural Tangent Kernel and Its Positivity
The Neural Tangent Kernel (NTK) is one powerful tool for analyzing the training dynamics of neural networks in the over-parameterized regime. Recently, the theoretical framework has been extended to physics-informed neural networks (PINNs) for solving linear PDEs, one highly popular class of neural PDE solvers. In the analysis, the positivity of the associated NTK plays a fundamental role. However, establishing the positivity of the NTK for PINNs is highly challenging, due to the presence of multiple differential operators. In this work, we propose a new theoretical framework, called Differential Neural Tangent Kernel (DNTK), for analyzing PINNs through the lens of the NTK, and establish the positivity of the infinite width DNTK for both shallow and deep neural networks for a wide class of activation functions, including RePU and smooth but non-polynomial activations, for all linear differential operators. These theoretical results lay the foundation for the analysis of gradient type algorithms for training PINNs.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
The Positivity of the Neural Tangent Kernel
The Neural Tangent Kernel (NTK) has emerged as a fundamental concept in the study of wide Neural Networks. In particular, it is known that the positivity of the NTK is directly related to the memorization capacity of suf…
MemorizationGradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
Recently, several studies have proven the global convergence and generalization abilities of the gradient descent method for two-layer ReLU networks. Most studies especially focused on the regression problems with the sq…
General ClassificationGeneralization BoundsAn Application of the Holonomic Gradient Method to the Neural Tangent Kernel
A holonomic system of linear partial differential equations is, roughly speaking, a system whose solution space is finite dimensional. A distribution that is a solution of a holonomic system is called a holonomic distrib…
Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operators
Spectral bias is a significant phenomenon in neural network training and can be explained by neural tangent kernel (NTK) theory. In this work, we develop the NTK theory for deep neural networks with physics-informed loss…
Is the neural tangent kernel of PINNs deep learning general partial differential equations always convergent ?
In this paper, we study the neural tangent kernel (NTK) for general partial differential equations (PDEs) based on physics-informed neural networks (PINNs). As we all know, the training of an artificial neural network ca…