Clip21: Error Feedback for Gradient Clipping

2023-05-30 · Sarit Khirirat, Eduard Gorbunov, Samuel Horváth, Rustem Islamov, Fakhri Karray, Peter Richtárik

Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipping, i.e., clipping applied to the gradients computed from local information at the nodes. While gradient clipping is an essential tool for injecting formal DP guarantees into gradient-based methods [1], it also induces bias which causes serious convergence issues specific to the distributed setting. Inspired by recent progress in the error-feedback literature which is focused on taming the bias/error introduced by communication compression operators such as Top-$k$ [2], and mathematical similarities between the clipping operator and contractive compression operators, we design Clip21 -- the first provably effective and practically useful error feedback mechanism for distributed methods with gradient clipping. We prove that our method converges at the same $\mathcal{O}\left(\frac{1}{K}\right)$ rate as distributed gradient descent in the smooth nonconvex regime, which improves the previous best $\mathcal{O}\left(\frac{1}{\sqrt{K}}\right)$ rate which was obtained under significantly stronger assumptions. Our method converges significantly faster in practice than competing methods.

📄 PDF Abstract BibTeX arXiv:2305.18929

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Gradient Clipping One difficulty that arises with optimization of deep neural networks is that large parameter gradients can lead an SGD optimizer to update…

Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach

2023-11-24 · Xinwei Zhang, Zhiqi Bu, Zhiwei Steven Wu, Mingyi Hong

Differentially Private Stochastic Gradient Descent with Gradient Clipping (DPSGD-GC) is a powerful tool for training deep learning models using sensitive data, providing both a solid theoretical privacy guarantee and hig…

Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy

2025-02-17 · Rustem Islamov, Samuel Horvath, Aurelien Lucchi, Peter Richtarik 외

Strong Differential Privacy (DP) and Optimization guarantees are two desirable properties for a method in Federated Learning (FL). However, existing algorithms do not achieve both properties at once: they either have opt…

Federated Learning

When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning

2026-02-11 · Mohammad Partohaghighi, Roummel Marcia, Bruce J. West, YangQuan Chen arxiv

Privacy-preserving training on sensitive data commonly relies on differentially private stochastic optimization with gradient clipping and Gaussian noise. The clipping threshold is a critical control knob: if set too sma…

Stochastic Optimization

Generalisation of RLHF under Reward Shift and Clipped KL Regularisation

2026-02-25 · Kenton Tang, Yuzhu Chen, Fengxiang He arxiv

Alignment and adaptation in large language models heavily rely on reinforcement learning from human feedback (RLHF); yet, theoretical understanding of its generalisability remains premature, especially when the learned r…

Reinforcement Learning

Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clipping

2024-06-04 · Lun Wang, Om Thakkar, Zhong Meng, Nicole Rafidi 외

Gradient clipping plays a vital role in training large-scale automatic speech recognition (ASR) models. It is typically applied to minibatch gradients to prevent gradient explosion, and to the individual sample gradients…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Memorization speech-recognition+1