On implicit regularization: Morse functions and applications to matrix factorization
In this paper, we revisit implicit regularization from the ground up using notions from dynamical systems and invariant subspaces of Morse functions. The key contributions are a new criterion for implicit regularization---a leading contender to explain the generalization power of deep models such as neural networks---and a general blueprint to study it. We apply these techniques to settle a conjecture on implicit regularization in matrix factorization.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
How regularization affects the geometry of loss functions
What neural networks learn depends fundamentally on the geometry of the underlying loss function. We study how different regularizers affect the geometry of this function. One of the most basic geometric properties of a …
Morse sequences
We introduce the notion of a Morse sequence, which provides a simple and effective approach to discrete Morse theory. A Morse sequence is a sequence composed solely of two elementary operations, that is, expansions (the …
Statistical Inference using the Morse-Smale Complex
The Morse-Smale complex of a function $f$ decomposes the sample space into cells where $f$ is increasing or decreasing. When applied to nonparametric density estimation and regression, it provides a way to represent, vis…
ClusteringDensity EstimationregressionLearning Stochastic Dynamical Systems as an Implicit Regularization with Graph Neural Networks
Stochastic Gumbel graph networks are proposed to learn high-dimensional time series, where the observed dimensions are often spatially correlated. To that end, the observed randomness and spatial-correlations are capture…
Time SeriesImplicit Regularization in Deep Matrix Factorization
Efforts to understand the generalization mystery in deep learning have led to the belief that gradient-based optimization induces a form of implicit regularization, a bias towards models of low "complexity." We study the…
Matrix Completion