How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysis
Domain generalization aims to learn invariance across multiple training domains, thereby enhancing generalization against out-of-distribution data. While gradient or representation matching algorithms have achieved remarkable success, these methods generally lack generalization guarantees or depend on strong assumptions, leaving a gap in understanding the underlying mechanism of distribution matching. In this work, we formulate domain generalization from a novel probabilistic perspective, ensuring robustness while avoiding overly conservative solutions. Through comprehensive information-theoretic analysis, we provide key insights into the roles of gradient and representation matching in promoting generalization. Our results reveal the complementary relationship between these two components, indicating that existing works focusing solely on either gradient or representation alignment are insufficient to solve the domain generalization problem. In light of these theoretical findings, we introduce IDM to simultaneously align the inter-domain gradients and representations. Integrated with the proposed PDM method for complex distribution matching, IDM achieves superior performance over various baseline methods.
Code (1)
Tasks
Domain GeneralizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers
Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming specialized training algorithms for domain generalization. The goa…
Domain AdaptationDomain GeneralizationOut-of-Distribution GeneralizationFederated Learning with Domain Generalization
Federated Learning (FL) enables a group of clients to jointly train a machine learning model with the help of a centralized server. Clients do not need to submit their local data to the server during training, and hence …
Domain GeneralizationFederated LearningDomain Generalizer: A Few-shot Meta Learning Framework for Domain Generalization in Medical Imaging
Deep learning models perform best when tested on target (test) data domains whose distribution is similar to the set of source (train) domains. However, model generalization can be hindered when there is significant diff…
Computed Tomography (CT)Domain GeneralizationFew-Shot LearningMeta-LearningDiversity Boosted Learning for Domain Generalization with Large Number of Domains
Machine learning algorithms minimizing the average training loss usually suffer from poor generalization performance due to the greedy exploitation of correlations among the training data, which are not stable under dist…
DiversityDomain GeneralizationPoint ProcessesRotated MNISTExactly Tight Information-Theoretic Generalization Error Bound for the Quadratic Gaussian Problem
We provide a new information-theoretic generalization error bound that is exactly tight (i.e., matching even the constant) for the canonical quadratic Gaussian (location) problem. Most existing bounds are order-wise loos…