Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective
We prove that the gradient descent training of a two-layer neural network on empirical or population risk may not decrease population risk at an order faster than $t^{-4/(d-2)}$ under mean field scaling. Thus gradient descent training for fitting reasonably smooth, but truly high-dimensional data may be subject to the curse of dimensionality. We present numerical evidence that gradient descent training with general Lipschitz target functions becomes slower and slower as the dimension increases, but converges at approximately the same rate in all dimensions when the target function lies in the natural function space for two-layer ReLU networks.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Does the Barron space really defy the curse of dimensionality?
The Barron space has become famous in the theory of (shallow) neural networks because it seemingly defies the curse of dimensionality. And while the Barron space (and generalizations) indeed defies (defy) the curse of di…
Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
We study the problem of estimating an unknown function from noisy data using shallow ReLU neural networks. The estimators we study minimize the sum of squared data-fitting errors plus a regularization term proportional t…
Quantifying the Empirical Wasserstein Distance to a Set of Measures: Beating the Curse of Dimensionality
We consider the problem of estimating the Wasserstein distance between the empirical measure and a set of probability measures whose expectations over a class of functions (hypothesis class) are constrained. If this clas…
Iterated Block Particle Filter for High-dimensional Parameter Learning: Beating the Curse of Dimensionality
Parameter learning for high-dimensional, partially observed, and nonlinear stochastic processes is a methodological challenge. Spatiotemporal disease transmission systems provide examples of such processes giving rise to…
State Space ModelsThe necessity of depth for artificial neural networks to approximate certain classes of smooth and bounded functions without the curse of dimensionality
In this article we study high-dimensional approximation capacities of shallow and deep artificial neural networks (ANNs) with the rectified linear unit (ReLU) activation. In particular, it is a key contribution of this w…