Smoothly Giving up: Robustness for Simple Models
There is a growing need for models that are interpretable and have reduced energy and computational cost (e.g., in health care analytics and federated learning). Examples of algorithms to train such models include logistic regression and boosting. However, one challenge facing these algorithms is that they provably suffer from label noise; this has been attributed to the joint interaction between oft-used convex loss functions and simpler hypothesis classes, resulting in too much emphasis being placed on outliers. In this work, we use the margin-based $\alpha$-loss, which continuously tunes between canonical convex and quasi-convex losses, to robustly train simple models. We show that the $\alpha$ hyperparameter smoothly introduces non-convexity and offers the benefit of "giving up" on noisy training examples. We also provide results on the Long-Servedio dataset for boosting and a COVID-19 survey dataset for logistic regression, highlighting the efficacy of our approach across multiple relevant domains.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningregressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GP-VAE: Deep Probabilistic Time Series Imputation
Multivariate time series with missing values are common in areas such as healthcare and finance, and have grown in number and complexity over the years. This raises the question whether deep learning methodologies can ou…
Deep LearningDimensionality ReductionImputationMissing Values+3Deep Amortized Variational Inference for Multivariate Time Series Imputation with Latent Gaussian Process Models
Multivariate time series with missing values are common in areas such as healthcare and finance, and have grown in number and complexity over the years. This raises the question whether deep learning methodologies can ou…
Deep LearningDimensionality ReductionImputationMissing Values+4On the Robustness of Neural Collapse and the Neural Collapse of Robustness
Neural Collapse refers to the curious phenomenon in the end of training of a neural network, where feature vectors and classification weights converge to a very simple geometrical arrangement (a simplex). While it has be…
Diffusion Operator Geometry of Feedforward Representations
Feedforward neural networks transform data through learned representations whose geometry shapes how classes separate and relate across successive layers. We study that geometry through diffusion operators. Each feature-…
Simple2Complex: Global Optimization by Gradient Descent
A method named simple2complex for modeling and training deep neural networks is proposed. Simple2complex train deep neural networks by smoothly adding more and more layers to the shallow networks, as the learning procedu…
global-optimization