Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variation
In this paper, we introduce the Hessian-Schatten total variation (HTV) -- a novel seminorm that quantifies the total "rugosity" of multivariate functions. Our motivation for defining HTV is to assess the complexity of supervised-learning schemes. We start by specifying the adequate matrix-valued Banach spaces that are equipped with suitable classes of mixed norms. We then show that the HTV is invariant to rotations, scalings, and translations. Additionally, its minimum value is achieved for linear mappings, which supports the common intuition that linear regression is the least complex learning model. Next, we present closed-form expressions of the HTV for two general classes of functions. The first one is the class of Sobolev functions with a certain degree of regularity, for which we show that the HTV coincides with the Hessian-Schatten seminorm that is sometimes used as a regularizer for image reconstruction. The second one is the class of continuous and piecewise-linear (CPWL) functions. In this case, we show that the HTV reflects the total change in slopes between linear regions that have a common facet. Hence, it can be viewed as a convex relaxation (l1-type) of the number of linear regions (l0-type) of CPWL mappings. Finally, we illustrate the use of our proposed seminorm.
Code (1)
Tasks
Image ReconstructionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generalized Hessian-Schatten Norm Regularization for Image Reconstruction
Regularization plays a crucial role in reliably utilizing imaging systems for scientific and medical investigations. It helps to stabilize the process of computationally undoing any degradation caused by physical limitat…
Image ReconstructionSecond-order optimization with lazy Hessians
We analyze Newton's method with lazy Hessian updates for solving general possibly non-convex optimization problems. We propose to reuse a previously seen Hessian for several iterations while computing new gradients at ea…
Image Restoration by Combined Order Regularization with Optimal Spatial Adaptation
Total Variation (TV) and related extensions have been popular in image restoration due to their robust performance and wide applicability. While the original formulation is still relevant after two decades of extensive r…
Image RestorationMRI ReconstructionProjected Hessian Learning: Fast Curvature Supervision for Accurate Machine-Learning Interatomic Potentials
The Hessian matrix (second derivatives) encodes far richer local curvature of the potential energy surface than energies and forces alone. However, training machine-learning interatomic potentials (MLIPs) with full Hessi…
What is the Inductive Bias of Flatness Regularization? A Study of Deep Matrix Factorization Models
Recent works on over-parameterized neural networks have shown that the stochasticity in optimizers has the implicit regularization effect of minimizing the sharpness of the loss function (in particular, the trace of its…