Sequential Compression Layers for Efficient Federated Learning in Foundational Models
Federated Learning (FL) has gained popularity for fine-tuning large language models (LLMs) across multiple nodes, each with its own private data. While LoRA has been widely adopted for parameter efficient federated fine-tuning, recent theoretical and empirical studies highlight its suboptimal performance in the federated learning context. In response, we propose a novel, simple, and more effective parameter-efficient fine-tuning method that does not rely on LoRA. Our approach introduces a small multi-layer perceptron (MLP) layer between two existing MLP layers the up proj (the FFN projection layer following the self-attention module) and down proj within the feed forward network of the transformer block. This solution addresses the bottlenecks associated with LoRA in federated fine tuning and outperforms recent LoRA-based approaches, demonstrating superior performance for both language models and vision encoders.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated Learningparameter-efficient fine-tuningSimilar Papers 제목 키워드 기반
Toward Efficient Federated Learning in Multi-Channeled Mobile Edge Network with Layerd Gradient Compression
A fundamental issue for federated learning (FL) is how to achieve optimal model performance under highly dynamic communication environments. This issue can be alleviated by the fact that modern edge devices usually can c…
Federated LearningEntropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devices
Federated learning (FL) over mobile and edge devices increasingly involves multimodal models in which clients differ in both sensing capability and computational capacity. Existing update compression schemes typically ap…
Federated LearningCommunication-Efficient Federated Learning via Quantized Compressed Sensing
In this paper, we present a communication-efficient federated learning framework inspired by quantized compressed sensing. The presented framework consists of gradient compression for wireless devices and gradient recons…
compressed sensingFederated LearningQuantizationFoundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights
Federated learning has the potential to unlock siloed data and distributed resources by enabling collaborative model training without sharing private data. As more complex foundational models gain widespread use, the nee…
Self-Supervised LearningFederated LearningTransfer LearningEdgeFLow: Serverless Federated Learning via Sequential Model Migration in Edge Networks
Federated Learning (FL) has emerged as a transformative distributed learning paradigm in the era of Internet of Things (IoT), reconceptualizing data processing methodologies. However, FL systems face significant communic…
Federated Learning