On the connections between algorithmic regularization and penalization for convex losses
In this work we establish the equivalence of algorithmic regularization and explicit convex penalization for generic convex losses. We introduce a geometric condition for the optimization path of a convex function, and show that if such a condition is satisfied, the optimization path of an iterative algorithm on the unregularized optimization problem can be represented as the solution path of a corresponding penalized problem.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Don't relax: early stopping for convex regularization
We consider the problem of designing efficient regularization algorithms when regularization is encoded by a (strongly) convex functional. Unlike classical penalization methods based on a relaxation approach, we propose …
Iterative Regularization for Learning with Convex Loss Functions
We consider the problem of supervised learning with convex loss functions and propose a new form of iterative regularization based on the subgradient method. Unlike other regularization approaches, in iterative regulariz…
BIG-bench Machine LearningGeneralized Preference Optimization: A Unified Approach to Offline Alignment
Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offlin…
Compound Poisson Processes, Latent Shrinkage Priors and Bayesian Nonconvex Penalization
In this paper we discuss Bayesian nonconvex penalization for sparse learning problems. We explore a nonparametric formulation for latent shrinkage parameters using subordinators which are one-dimensional L\'{e}vy process…
regressionSparse LearningAccelerated Gradient Flow: Risk, Stability, and Implicit Regularization
Acceleration and momentum are the de facto standard in modern applications of machine learning and optimization, yet the bulk of the work on implicit regularization focuses instead on unaccelerated methods. In this paper…
regression