On Connected Sublevel Sets in Deep Learning
This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large global valley.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningSimilar Papers 제목 키워드 기반
A Note on Connectivity of Sublevel Sets in Deep Learning
It is shown that for deep neural networks, a single wide layer of width $N+1$ ($N$ being the number of training samples) suffices to prove the connectivity of sublevel sets of the training loss function. In the two-layer…
Deep LearningLow-loss connection of weight vectors: distribution-based approaches
Recent research shows that sublevel sets of the loss surfaces of overparameterized networks are connected, exactly or approximately. We describe and compare experimentally a panel of methods used to connect two low-loss …
SensitivityGuarantees for Hierarchical Clustering by the Sublevel Set method
Meila (2018) introduces an optimization based method called the Sublevel Set method, to guarantee that a clustering is nearly optimal and "approximately correct" without relying on any assumptions about the distribution …
ClusteringRobust Control Barrier Functions under High Relative Degree and Input Constraints for Satellite Trajectories
This paper presents methodologies for constructing Control Barrier Functions (CBFs) for nonlinear, control-affine systems, in the presence of input constraints and bounded disturbances. More specifically, given a constra…
How to Learn a Star: Binary Classification with Starshaped Polyhedral Sets
We consider binary classification restricted to a class of continuous piecewise linear functions whose decision boundaries are (possibly nonconvex) starshaped polyhedral sets, supported on a fixed polyhedral simplicial f…
Binary Classification