Generalization vs. Specialization under Concept Shift
Machine learning models are often brittle under distribution shift, i.e., when data distributions at test time differ from those during training. Understanding this failure mode is central to identifying and mitigating safety risks of mass adoption of machine learning. Here we analyze ridge regression under concept shift -- a form of distribution shift in which the input-label relationship changes at test time. We derive an exact expression for prediction risk in the high-dimensional limit. Our results reveal nontrivial effects of concept shift on generalization performance, depending on the properties of robust and nonrobust features of the input. We show that test performance can exhibit a nonmonotonic data dependence, even when double descent is absent. Finally, our experiments on MNIST and FashionMNIST suggest that this intriguing behavior is present also in classification problems.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models
Recent empirical studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT), and have found it to yield significant performance improvements. However, t…
Economic Rationality under Specialization: Evidence of Decision Bias in AI Agents
In the study by Chen et al. (2023) [01], the large language model GPT demonstrated economic rationality comparable to or exceeding the average human level in tasks such as budget allocation and risk preference. Building …
Decision MakingLanguage ModelingLanguage ModellingLarge Language ModelDecoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
Federated Learning (FL) with pre-trained Vision-Language Models (VLMs) has emerged as a promising paradigm for various downstream tasks. By leveraging its strong representations, recent studies improve task adaptation un…
Reinforcement LearningFederated LearningNICO++: Towards Better Benchmarking for Domain Generalization
Despite the remarkable performance that modern deep neural networks have achieved on independent and identically distributed (I.I.D.) data, they can crash under distribution shifts. Most current evaluation methods for do…
BenchmarkingDomain GeneralizationGeneralization BoundsModel SelectionSTAR: Rethinking MoE Routing as Structure-Aware Subspace Learning
Mixture-of-Experts (MoE) scales model capacity efficiently by selectively routing inputs to a specialized subset of experts. However, input-expert specialization, the core motivation of MoE, critically depends on whether…