The Effect of the Intrinsic Dimension on the Generalization of Quadratic Classifiers
It has been recently observed that neural networks, unlike kernel methods, enjoy a reduced sample complexity when the distribution is isotropic (i.e., when the covariance matrix is the identity). We find that this sensitivity to the data distribution is not exclusive to neural networks, and the same phenomenon can be observed on the class of quadratic classifiers (i.e., the sign of a quadratic polynomial) with a nuclear-norm constraint. We demonstrate this by deriving an upper bound on the Rademacher Complexity that depends on two key quantities: (i) the intrinsic dimension, which is a measure of isotropy, and (ii) the largest eigenvalue of the second moment (covariance) matrix of the distribution. Our result improves the dependence on the dimension over the best previously known bound and precisely quantifies the relation between the sample complexity and the level of isotropy of the distribution.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
High-dimensional quadratic classifiers in non-sparse settings
We consider high-dimensional quadratic classifiers in non-sparse settings. The target of classification rules is not Bayes error rates in the context. The classifier based on the Mahalanobis distance does not always give…
feature selectionGeneral ClassificationVocal Bursts Intensity PredictionExploiting the Layered Intrinsic Dimensionality of Deep Models for Practical Adversarial Training
Despite being a heavily researched topic, Adversarial Training (AT) is rarely, if ever, deployed in practical AI systems for two primary reasons: (i) the gained robustness is frequently accompanied by a drop in generaliz…
DecoderGPURAGOne Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks
Inspired by neuronal diversity in the biological neural system, a plethora of studies proposed to design novel types of artificial neurons and introduce neuronal diversity into artificial neural networks. Recently propos…
DiversityKernel-Free Universum Quadratic Surface Twin Support Vector Machines for Imbalanced Data
Binary classification tasks with imbalanced classes pose significant challenges in machine learning. Traditional classifiers often struggle to accurately capture the characteristics of the minority class, resulting in bi…
Binary ClassificationClassificationimbalanced classificationLocal Intrinsic Dimension of Representations Predicts Alignment and Generalization in AI Models and Human Brain
Recent work has found that neural networks with stronger generalization tend to exhibit higher representational alignment with one another across architectures and training paradigms. In this work, we show that models wi…