Cauchy-Schwarz Regularizers
We introduce a novel class of regularization functions, called Cauchy-Schwarz (CS) regularizers, which can be designed to induce a wide range of properties in solution vectors of optimization problems. To demonstrate the versatility of CS regularizers, we derive regularization functions that promote discrete-valued vectors, eigenvectors of a given matrix, and orthogonal matrices. The resulting CS regularizers are simple, differentiable, and can be free of spurious stationary points, making them suitable for gradient-based solvers and large-scale optimization problems. In addition, CS regularizers automatically adapt to the appropriate scale, which is, for example, beneficial when discretizing the weights of neural networks. To demonstrate the efficacy of CS regularizers, we provide results for solving underdetermined systems of linear equations and weight quantization in neural networks. Furthermore, we discuss specializations, variations, and generalizations, which lead to an even broader class of new and possibly more powerful regularizers.
Code (1)
Tasks
QuantizationSimilar Papers 제목 키워드 기반
Real Vector Spaces and the Cauchy-Schwarz Inequality in ACL2(r)
We present a mechanical proof of the Cauchy-Schwarz inequality in ACL2(r) and a formalisation of the necessary mathematics to undertake such a proof. This includes the formalisation of $\mathbb{R}^n$ as an inner product …
Cauchy-Schwarz Fairness Regularizer
Group fairness in machine learning is often enforced by adding a regularizer that reduces the dependence between model predictions and sensitive attributes. However, existing regularizers are built on heterogeneous dista…
Efficient Sensor Management for Multitarget Tracking in Passive Sensor Networks via Cauchy-Schwarz Divergence
This paper presents an efficient sensor management approach for multi-target tracking in passive sensor networks. Compared with active sensor networks, passive sensor networks have larger uncertainty due to the nature of…
ManagementCauchy-Schwarz Divergence Information Bottleneck for Regression
The information bottleneck (IB) approach is popular to improve the generalization, robustness and explainability of deep neural networks. Essentially, it aims to find a minimum sufficient representation $\mathbf{t}$ by s…
Adversarial RobustnessInformation PlaneregressionVariational InferenceOn Hölder projective divergences
We describe a framework to build distances by measuring the tightness of inequalities, and introduce the notion of proper statistical divergences and improper pseudo-divergences. We then consider the H\"older ordinary an…
Clustering