LASER: Linear Compression in Wireless Distributed Optimization
Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this paper, we close this gap and introduce LASER: LineAr CompreSsion in WirEless DistRibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain $50$-$64 \%$ improvement in perplexity over our baselines for noisy channels.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributed OptimizationLanguage ModelingLanguage ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Artificial Neural Network for Resource Allocation in Laser-based Optical wireless Networks
Optical wireless communication offers unprecedented communication speeds that can support the massive use of the Internet on a daily basis. In indoor environments, optical wireless networks are usually multi-user multipl…
Dataset GenerationInnovation Compression for Communication-efficient Distributed Optimization with Linear Convergence
Information compression is essential to reduce communication cost in distributed optimization over peer-to-peer networks. This paper proposes a communication-efficient linearly convergent distributed (COLD) algorithm to …
Distributed OptimizationResource Allocation in Laser-based Optical Wireless Cellular Networks
Optical wireless communication provides data transmission at high speeds which can satisfy the increasing demands for connecting a massive number of devices to the Internet. In this paper, vertical-cavity surface-emittin…
Distributed and Stochastic Optimization Methods with Gradient Compression and Local Steps
In this thesis, we propose new theoretical frameworks for the analysis of stochastic and distributed methods with error compensation and local updates. Using these frameworks, we develop more than 20 new optimization met…
Stochastic OptimizationJoint Collaboration and Compression Design for Distributed Sequential Estimation in a Wireless Sensor Network
In this work, we propose a joint collaboration-compression framework for sequential estimation of a random vector parameter in a resource constrained wireless sensor network (WSN). Specifically, we propose a framework wh…