Implicit Regularization in Matrix Factorization
We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix $X$ with gradient descent on a factorization of $X$. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Implicit Regularization in Deep Matrix Factorization
Efforts to understand the generalization mystery in deep learning have led to the belief that gradient-based optimization induces a form of implicit regularization, a bias towards models of low "complexity." We study the…
Matrix CompletionOn implicit regularization: Morse functions and applications to matrix factorization
In this paper, we revisit implicit regularization from the ground up using notions from dynamical systems and invariant subspaces of Morse functions. The key contributions are a new criterion for implicit regularization-…
Implicit Regularization in Deep Learning May Not Be Explainable by Norms
Mathematically characterizing the implicit regularization induced by gradient-based optimization is a longstanding pursuit in the theory of deep learning. A widespread hope is that a characterization based on minimizatio…
Deep LearningMatrix CompletionOpen-Ended Question AnsweringConnectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion
Matrix factorization models have been extensively studied as a valuable test-bed for understanding the implicit biases of overparameterized models. Although both low nuclear norm and low rank regularization have been stu…
Matrix CompletionImplicit Regularization with Polynomial Growth in Deep Tensor Factorization
We study the implicit regularization effects of deep learning in tensor factorization. While implicit regularization in deep matrix and 'shallow' tensor factorization via linear and certain type of non-linear neural netw…