On how to avoid exacerbating spurious correlations when models are overparameterized
Overparameterized models fail to generalize well in the presence of data imbalance even when combined with traditional techniques for mitigating imbalances. This paper focuses on imbalanced classification datasets, in which a small subset of the population -- a minority -- may contain features that correlate spuriously with the class label. For a parametric family of cross-entropy loss modifications and a representative Gaussian mixture model, we derive non-asymptotic generalization bounds on the worst-group error that shed light on the role of different hyper-parameters. Specifically, we prove that, when appropriately tuned, the recently proposed VS-loss learns a model that is fair towards minorities even when spurious features are strong. On the other hand, alternative heuristics, such as the weighted CE and the LA-loss, can fail dramatically. Compared to previous works, our bounds hold for more general models, they are non-asymptotic, and, they apply even at scenarios of extreme imbalance.
Code (0)
등록된 구현이 없습니다.
Tasks
Generalization Boundsimbalanced classificationSimilar Papers 제목 키워드 기반
Overparameterization hurts worst-group accuracy with spurious correlations
Increasing model capacity well beyond the point of zero training error has been observed to improve average test accuracy. However, such overparameterized models have been recently shown to obtain low worst-group accurac…
An Investigation of Why Overparameterization Exacerbates Spurious Correlations
We study why overparameterization -- increasing model size well beyond the point of zero training error -- can hurt test error on minority groups despite improving average test error when there are spurious correlations …
Inductive BiasGenerative Classifiers Avoid Shortcut Solutions
Discriminative approaches to classification often learn shortcuts that hold in-distribution but fail even under minor distribution shift. This failure mode stems from an overreliance on features that are spuriously corre…
The Impact of Coreset Selection on Spurious Correlations and Group Robustness
Coreset selection methods have shown promise in reducing the training data size while maintaining model performance for data-efficient machine learning. However, as many datasets suffer from biases that cause models to l…
UnLearning from Experience to Avoid Spurious Correlations
While deep neural networks can achieve state-of-the-art performance in many tasks, these models are more fragile than they appear. They are prone to learning spurious correlations in their training data, leading to surpr…