Connecting Optimization and Regularization Paths
We study the implicit regularization properties of optimization techniques by explicitly connecting their optimization paths to the regularization paths of ``corresponding'' regularized problems. This surprising connection shows that iterates of optimization techniques such as gradient descent and mirror descent are \emph{pointwise} close to solutions of appropriately regularized objectives. While such a tight connection between optimization and regularization is of independent intellectual interest, it also has important implications for machine learning: we can port results from regularized estimators to optimization, and vice versa. We investigate one key consequence, that borrows from the well-studied analysis of regularized estimators, to then obtain tight excess risk bounds of the iterates generated by optimization techniques.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Implicit Regularization Paths of Weighted Neural Representations
We study the implicit regularization effects induced by (observation) weighting of pretrained features. For weight and feature matrices of bounded operator norms that are infinitesimally free with respect to (normalized)…
A multiobjective continuation method to compute the regularization path of deep neural networks
Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability of models (due to the smaller number of relevant features), and robustness. For lin…
Multiobjective OptimizationUnpaired Image-to-Image Translation With Shortest Path Regularization
Unpaired image-to-image translation aims to learn proper mappings that can map images from one domain to another domain while preserving the content of the input image. However, with large enough capacities, the netw…
Image-to-Image TranslationTranslationAn Optimization Algorithm for Customer Topological Paths Identification in Electrical Distribution Networks
A customer topological path represents the sequence of network elements connecting an MV/LV transformer to a customer. Accurate knowledge of these paths is crucial for distribution system operators (DSOs) in digitalizati…
Capacity Control of ReLU Neural Networks by Basis-path Norm
Recently, path norm was proposed as a new capacity measure for neural networks with Rectified Linear Unit (ReLU) activation function, which takes the rescaling-invariant property of ReLU into account. It has been shown t…