Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing
As training datasets grow larger, we aspire to develop models that generalize well to any diverse test distribution, even if the latter deviates significantly from the training data. Various approaches like domain adaptation, domain generalization, and robust optimization attempt to address the out-of-distribution challenge by posing assumptions about the relation between training and test distribution. Differently, we adopt a more conservative perspective by accounting for the worst-case error across all sufficiently diverse test distributions within a known domain. Our first finding is that training on a uniform distribution over this domain is optimal. We also interrogate practical remedies when uniform samples are unavailable by considering methods for mitigating non-uniformity through finetuning and rebalancing. Our theory provides a mathematical grounding for previous observations on the role of entropy and rebalancing for o.o.d. generalization and foundation model training. We also provide new empirical evidence across tasks involving o.o.d. shifts which illustrate the broad applicability of our perspective.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationDomain GeneralizationSimilar Papers 제목 키워드 기반
GENTLE: A Genre-Diverse Multilayer Challenge Set for English NLP and Linguistic Evaluation
We present GENTLE, a new mixed-genre English challenge corpus totaling 17K tokens and consisting of 8 unusual text types for out-of domain evaluation: dictionary entries, esports commentaries, legal documents, medical no…
coreference-resolutionCoreference ResolutionDependency ParsingDiscourse Parsing+1Learning Dynamics of VLM Finetuning
Preference-based finetuning of vision--language models (VLMs) is brittle: trivially wrong negatives inject uninformative gradients that destabilize training. We recast alignment as \textbf{learning-dynamics--aware optimi…
RAU: Towards Regularized Alignment and Uniformity for Representation Learning in Recommendation
Recommender systems (RecSys) have become essential in modern society, driving user engagement and satisfaction across diverse online platforms. Most RecSys focuses on designing a powerful encoder to embed users and items…
Recommendation SystemsRepresentation LearningGeneralizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap
The neural collapse (NC) phenomenon describes an underlying geometric symmetry for deep neural networks, where both deeply learned features and classifiers converge to a simplex equiangular tight frame. It has been shown…