Federated Learning with Layer Skipping: Efficient Training of Large Language Models for Healthcare NLP
Federated learning (FL) enables collaborative model training across organizations without sharing raw data, addressing crucial privacy concerns in healthcare natural language processing (NLP). However, training large language models (LLMs) in federated settings faces significant challenges, including communication overhead and data heterogeneity. We propose Layer-Skipping Federated Learning, where only selected layers of a pre-trained LLM are fine-tuned across clients while others remain frozen. Applied to LLaMA 3.2-1B, our approach reduces communication costs by approximately 70% while maintaining performance within 2% of centralized training. We evaluate our method on clinical NER and classification tasks using i2b2 and MIMIC-III datasets. Our experiments demonstrate that Layer-Skipping FL outperforms competitive baselines, handles non-IID clinical data distributions effectively, and shows robustness when combined with differential privacy. This approach represents a practical solution for privacy-preserving collaborative learning in healthcare NLP.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningNERPrivacy PreservingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
L-FGADMM: Layer-Wise Federated Group ADMM for Communication Efficient Decentralized Deep Learning
This article proposes a communication-efficient decentralized deep learning algorithm, coined layer-wise federated group ADMM (L-FGADMM). To minimize an empirical risk, every worker in L-FGADMM periodically communicates …
Federated LearningSkipping Computations in Multimodal LLMs
Large Language Models (LLMs) have demonstrated remarkable success in both textual and multimodal domains. However, this success often comes with substantial computational costs, particularly when handling lengthy sequenc…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy
Recently, dynamic computation methods have shown notable acceleration for Large Language Models (LLMs) by skipping several layers of computations through elaborate heuristics or additional predictors. However, in the dec…
Machine TranslationText SummarizationAdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference
Long-context large language models (LLMs) inference is increasingly critical, motivating a number of studies devoted to alleviating the substantial storage and computational costs in such scenarios. Layer-wise skipping m…
SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks
Data and pipeline parallelism are ubiquitous for training of Large Language Models (LLM) on distributed nodes. Driven by the need for cost-effective training, recent work explores efficient communication arrangement for …
Scheduling