Federated Neural Compression Under Heterogeneous Data
We discuss a federated learned compression problem, where the goal is to learn a compressor from real-world data which is scattered across clients and may be statistically heterogeneous, yet share a common underlying representation. We propose a distributed source model that encompasses both characteristics, and naturally suggests a compressor architecture that uses analysis and synthesis transforms shared by clients. Inspired by personalized federated learning methods, we employ an entropy model that is personalized to each client. This allows for a global latent space to be learned across clients, and personalized entropy models that adapt to the clients' latent distributions. We show empirically that this strategy outperforms solely local methods, which indicates that learned compression also benefits from a shared global representation in statistically heterogeneous federated settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningPersonalized Federated LearningSimilar Papers 제목 키워드 기반
Entropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devices
Federated learning (FL) over mobile and edge devices increasingly involves multimodal models in which clients differ in both sensing capability and computational capacity. Existing update compression schemes typically ap…
Federated LearningFedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization
One underlying assumption of recent federated learning (FL) paradigms is that all local models usually share the same network architecture and size, which becomes impractical for devices with different hardware resources…
Distributed ComputingFederated LearningLow-rank compressionMatrix Factorization / Decomposition+1Heterogeneity-Aware Cooperative Federated Edge Learning with Adaptive Computation and Communication Compression
Motivated by the drawbacks of cloud-based federated learning (FL), cooperative federated edge learning (CFEL) has been proposed to improve efficiency for FL over mobile edge networks, where multiple edge servers collabor…
Federated LearningCA-HFP: Curvature-Aware Heterogeneous Federated Pruning with Model Reconstruction
Federated learning on heterogeneous edge devices requires personalized compression while preserving aggregation compatibility and stable convergence. We present Curvature-Aware Heterogeneous Federated Pruning (CA-HFP), a…
Federated LearningBandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, these methods encounter challenges such as t…
Data CompressionFederated Learning