A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern
The work "Loss Landscape Sightseeing with Multi-Point Optimization" (Skorokhodov and Burtsev, 2019) demonstrated that one can empirically find arbitrary 2D binary patterns inside loss surfaces of popular neural networks. In this paper we prove that: (i) this is a general property of deep universal approximators; and (ii) this property holds for arbitrary smooth patterns, for other dimensionalities, for every dataset, and any neural network that is sufficiently deep and wide. Our analysis predicts not only the existence of all such low-dimensional patterns, but also two other properties that were observed empirically: (i) that it is easy to find these patterns; and (ii) that they transfer to other data-sets (e.g. a test-set).
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Loss Landscape Sightseeing with Multi-Point Optimization
We present multi-point optimization: an optimization technique that allows to train several models simultaneously without the need to keep the parameters of each one individually. The proposed method is used for a thorou…
Understanding Local Minima in Neural Networks by Loss Surface Decomposition
To provide principled ways of designing proper Deep Neural Network (DNN) models, it is essential to understand the loss surface of DNNs under realistic assumptions. We introduce interesting aspects for understanding the …
Piecewise Strong Convexity of Neural Networks
We study the loss surface of a feed-forward neural network with ReLU non-linearities, regularized with weight decay. We show that the regularized loss function is piecewise strongly convex on an important open set which …
image-classificationImage ClassificationLearning TheoryA large language model-type architecture for high-dimensional molecular potential energy surfaces
Computing high dimensional potential surfaces for molecular and materials systems is considered to be a great challenge in computational chemistry with potential impact in a range of areas including fundamental predictio…
Computational chemistryLanguage ModelingLanguage ModellingLarge Language ModelNatural Language Generation by Hierarchical Decoding with Linguistic Patterns
Natural language generation (NLG) is a critical component in spoken dialogue systems. Classic NLG can be divided into two phases: (1) sentence planning: deciding on the overall sentence structure, (2) surface realization…
DecoderSentenceSpoken Dialogue SystemsText Generation