A Note on Connectivity of Sublevel Sets in Deep Learning
It is shown that for deep neural networks, a single wide layer of width $N+1$ ($N$ being the number of training samples) suffices to prove the connectivity of sublevel sets of the training loss function. In the two-layer setting, the same property may not hold even if one has just one neuron less (i.e. width $N$ can lead to disconnected sublevel sets).
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningSimilar Papers 제목 키워드 기반
From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks
We study pathwise connectivity of sublevel sets for one-hidden-layer ReLU networks with constrained first-layer weights and an $\ell_1$ penalty on the output layer. The data term is assumed convex and globally Lipschitz …
On Connected Sublevel Sets in Deep Learning
This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad…
Deep LearningWhen Are Solutions Connected in Deep Networks?
The question of how and why the phenomenon of mode connectivity occurs in training deep neural networks has gained remarkable attention in the research community. From a theoretical perspective, two possible explanations…
Guarantees for Hierarchical Clustering by the Sublevel Set method
Meila (2018) introduces an optimization based method called the Sublevel Set method, to guarantee that a clustering is nearly optimal and "approximately correct" without relying on any assumptions about the distribution …
ClusteringRobust Control Barrier Functions under High Relative Degree and Input Constraints for Satellite Trajectories
This paper presents methodologies for constructing Control Barrier Functions (CBFs) for nonlinear, control-affine systems, in the presence of input constraints and bounded disturbances. More specifically, given a constra…