Robust Generalization despite Distribution Shift via Minimum Discriminating Information
Training models that perform well under distribution shifts is a central challenge in machine learning. In this paper, we introduce a modeling framework where, in addition to training data, we have partial structural knowledge of the shifted test distribution. We employ the principle of minimum discriminating information to embed the available prior knowledge, and use distributionally robust optimization to account for uncertainty due to the limited samples. By leveraging large deviation results, we obtain explicit generalization bounds with respect to the unknown shifted distribution. Lastly, we demonstrate the versatility of our framework by demonstrating it on two rather distinct applications: (1) training classifiers on systematically biased data and (2) off-policy evaluation in Markov Decision Processes.
Code (2)
Tasks
Generalization BoundsOff-policy evaluationSimilar Papers 제목 키워드 기반
Memory Classifiers: Two-stage Classification for Robustness in Machine Learning
The performance of machine learning models can significantly degrade under distribution shifts of the data. We propose a new method for classification which can improve robustness to distribution shifts, by combining exp…
BIG-bench Machine LearningClassificationData AugmentationGeneralization Bounds+1Generalization of Scaled Deep ResNets in the Mean-Field Regime
Despite the widespread empirical success of ResNet, the generalization properties of deep ResNet are rarely explored beyond the lazy training regime. In this work, we investigate \emph{scaled} ResNet in the limit of infi…
Generalization BoundsTrained Transformers Learn Linear Models In-Context
Attention-based neural networks such as transformers have demonstrated a remarkable ability to exhibit in-context learning (ICL): Given a short prompt sequence of tokens from an unseen task, they can formulate relevant p…
In-Context LearningregressionTowards In-distribution Compatibility in Out-of-distribution Detection
Deep neural network, despite its remarkable capability of discriminating targeted in-distribution samples, shows poor performance on detecting anomalous out-of-distribution data. To address this defect, state-of-the-art …
Out-of-Distribution DetectionDomain-wise Adversarial Training for Out-of-Distribution Generalization
Despite the impressive success on many tasks, deep learning models are shown to rely on spurious features, which will catastrophically fail when generalized to out-of-distribution (OOD) data. To alleviate this issue, Inv…
DiversityOut-of-Distribution Generalization