Exploiting Personalized Invariance for Better Out-of-distribution Generalization in Federated Learning

2022-11-21 · Xueyang Tang, Song Guo, Jie Zhang

Recently, data heterogeneity among the training datasets on the local clients (a.k.a., Non-IID data) has attracted intense interest in Federated Learning (FL), and many personalized federated learning methods have been proposed to handle it. However, the distribution shift between the training dataset and testing dataset on each client is never considered in FL, despite it being general in real-world scenarios. We notice that the distribution shift (a.k.a., out-of-distribution generalization) problem under Non-IID federated setting becomes rather challenging due to the entanglement between personalized and spurious information. To tackle the above problem, we elaborate a general dual-regularized learning framework to explore the personalized invariance, compared with the exsiting personalized federated learning methods which are regularized by a single baseline (usually the global model). Utilizing the personalized invariant features, the developed personalized models can efficiently exploit the most relevant information and meanwhile eliminate spurious information so as to enhance the out-of-distribution generalization performance for each client. Both the theoretical analysis on convergence and OOD generalization performance and the results of extensive experiments demonstrate the superiority of our method over the existing federated learning and invariant learning methods, in diverse out-of-distribution and Non-IID data cases.

📄 PDF Abstract BibTeX arXiv:2211.11243

Code (0)

등록된 구현이 없습니다.

Tasks

Federated LearningOut-of-Distribution GeneralizationPersonalized Federated Learning

Effect-Invariant Mechanisms for Policy Generalization

2023-06-19 · Sorawit Saengkyongam, Niklas Pfister, Predrag Klasnja, Susan Murphy 외

Policy learning is an important component of many real-world learning systems. A major challenge in policy learning is how to adapt efficiently to unseen environments or tasks. Recently, it has been suggested to exploit …

On the Benefits of Invariance in Neural Networks

2020-05-01 · Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal 외

Many real world data analysis problems exhibit invariant structure, and models that take advantage of this structure have shown impressive empirical performance, particularly in deep learning. While the literature contai…

Data Augmentation

On the Strong Correlation Between Model Invariance and Generalization

2022-07-14 · Weijian Deng, Stephen Gould, Liang Zheng

Generalization and invariance are two essential properties of any machine learning model. Generalization captures a model's ability to classify unseen data while invariance measures consistency of model predictions on tr…

Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization

2021-06-11 · NeurIPS 2021 12 · Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet 외

The invariance principle from causality is at the heart of notable approaches such as invariant risk minimization (IRM) that seek to address out-of-distribution (OOD) generalization failures. Despite the promising theory…

Out-of-Distribution Generalization regression

Cross-Domain Causal Preference Learning for Out-of-Distribution Recommendation

2024-04-23 · Zhuhang Li, Ning Yang

Recommender systems use users' historical interactions to learn their preferences and deliver personalized recommendations from a vast array of candidate items. Current recommender systems primarily rely on the assumptio…

Out-of-Distribution Generalization Recommendation Systems