Importance Tempering: Group Robustness for Overparameterized Models
Although overparameterized models have shown their success on many machine learning tasks, the accuracy could drop on the testing distribution that is different from the training one. This accuracy drop still limits applying machine learning in the wild. At the same time, importance weighting, a traditional technique to handle distribution shifts, has been demonstrated to have less or even no effect on overparameterized models both empirically and theoretically. In this paper, we propose importance tempering to improve the decision boundary and achieve consistently better results for overparameterized models. Theoretically, we justify that the selection of group temperature can be different under label shift and spurious correlation setting. At the same time, we also prove that properly selected temperatures can extricate the minority collapse for imbalanced classification. Empirically, we achieve state-of-the-art results on worst group classification tasks using importance tempering.
Code (0)
등록된 구현이 없습니다.
Tasks
imbalanced classificationSimilar Papers 제목 키워드 기반
Sharp analysis of out-of-distribution error for "importance-weighted" estimators in the overparameterized regime
Overparameterized models that achieve zero training error are observed to generalize well on average, but degrade in performance when faced with data that is under-represented in the training sample. In this work, we stu…
Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
Overparameterized neural networks can be highly accurate on average on an i.i.d. test set yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that hold on average but not in such…
Domain GeneralizationNatural Language InferenceOut-of-Distribution GeneralizationStochastic OptimizationDISTRIBUTIONALLY ROBUST NEURAL NETWORKS FOR GROUP SHIFTS ON THE IMPORTANCE OF REGULARIZATION FOR WORST-CASE GENERALIZATION
Overparameterized neural networks can be highly accurate on average on an i.i.d. test set yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that hold on average but not in such…
Natural Language InferenceStochastic OptimizationFlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Tempering
Multi-modal and high-dimensional posteriors present significant challenges for variational inference, causing mode-seeking behavior and collapse despite the theoretical expressiveness of normalizing flows. Traditional an…
Variational InferenceScalable Importance Tempering and Bayesian Variable Selection
We propose a Monte Carlo algorithm to sample from high dimensional probability distributions that combines Markov chain Monte Carlo and importance sampling. We provide a careful theoretical analysis, including guarantees…
Variable Selection