Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks

2024-01-18 · Yichao Du, Zhirui Zhang, Linan Yue, Xu Huang, Yuqing Zhang, Tong Xu, Linli Xu, Enhong Chen

To protect privacy and meet legal regulations, federated learning (FL) has gained significant attention for training speech-to-text (S2T) systems, including automatic speech recognition (ASR) and speech translation (ST). However, the commonly used FL approach (i.e., \textsc{FedAvg}) in S2T tasks typically suffers from extensive communication overhead due to multi-round interactions based on the whole model and performance degradation caused by data heterogeneity among clients.To address these issues, we propose a personalized federated S2T framework that introduces \textsc{FedLoRA}, a lightweight LoRA module for client-side tuning and interaction with the server to minimize communication overhead, and \textsc{FedMem}, a global model equipped with a $k$-nearest-neighbor ($k$NN) classifier that captures client-specific distributional shifts to achieve personalization and overcome data heterogeneity. Extensive experiments based on Conformer and Whisper backbone models on CoVoST and GigaSpeech benchmarks show that our approach significantly reduces the communication overhead on all S2T tasks and effectively personalizes the global model to overcome data heterogeneity.

📄 PDF Abstract BibTeX arXiv:2401.10070

Code (0)

등록된 구현이 없습니다.

Tasks

Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Federated Learning Personalized Federated Learning speech-recognition Speech RecognitionSpeech-to-Text

Fed-PISA: Federated Voice Cloning via Personalized Identity-Style Adaptation

2025-09-19 · Qi Wang, Shituo Ma, Guoxin Yu, Hanyang Peng 외 arxiv

Voice cloning for Text-to-Speech (TTS) aims to generate expressive and personalized speech from text using limited data from a target speaker. Federated Learning (FL) offers a collaborative and privacy-preserving framewo…

Collaborative Filtering Federated Learning

Decoupled Federated Learning for ASR with Non-IID Data

2022-06-18 · Han Zhu, Jindong Wang, Gaofeng Cheng, Pengyuan Zhang 외

Automatic speech recognition (ASR) with federated learning (FL) makes it possible to leverage data from multiple clients without compromising privacy. The quality of FL-based ASR could be measured by recognition performa…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Federated Learning speech-recognition+1

Towards Federated Graph Learning in One-shot Communication

2024-11-18 · Guochen Yan, Xunkai Li, Luyuan Xie, Wentao Zhang 외

Federated Graph Learning (FGL) has emerged as a promising paradigm for breaking data silos among distributed private graphs. In practical scenarios involving heterogeneous distributed graph data, personalized Federated G…

Federated Learning Graph Learning Node Classification

Communication-Efficient Personalized Adaptation via Federated-Local Model Merging

2026-02-20 · Yinan Zou, Md Kamran Chowdhury Shisher, Christopher G. Brinton, Vishrant Tripathi arxiv

Parameter-efficient fine-tuning methods, such as LoRA, offer a practical way to adapt large vision and language models to client tasks. However, this becomes particularly challenging under task-level heterogeneity in fed…

parameter-efficient fine-tuning General Knowledge

Federated Marginal Personalization for ASR Rescoring

2020-12-01 · Zhe Liu, Fuchun Peng

We introduce federated marginal personalization (FMP), a novel method for continuously updating personalized neural network language models (NNLMs) on private devices using federated learning (FL). Instead of fine-tuning…

Federated Learning speech-recognition Speech Recognition