Layer-wise Quantization for Quantized Optimistic Dual Averaging
Modern deep neural networks exhibit heterogeneity across numerous layers of various types such as residuals, multi-head attention, etc., due to varying structures (dimensions, activation functions, etc.), distinct representation characteristics, which impact predictions. We develop a general layer-wise quantization framework with tight variance and code-length bounds, adapting to the heterogeneities over the course of training. We then apply a new layer-wise quantization technique within distributed variational inequalities (VIs), proposing a novel Quantized Optimistic Dual Averaging (QODA) algorithm with adaptive learning rates, which achieves competitive convergence rates for monotone VIs. We empirically show that QODA achieves up to a $150\%$ speedup over the baselines in end-to-end training time for training Wasserstein GAN on $12+$ GPUs.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationSimilar Papers 제목 키워드 기반
CALM: A CKA-Guided Adaptive Layer-Wise Modularization Framework for LLM Quantization
Current mainstream post-training quantization methods for large language models typically apply a uniform quantization strategy across all network layers, overlooking the substantial differences in algorithmic suitabilit…
Learning to Quantize Deep Neural Networks: A Competitive-Collaborative Approach
Reducing the model size and computation costs for dedicated AI accelerator designs, neural network quantization methods have at- tracted momentous attention recently. Unfortunately, merely minimizing quantization loss…
QuantizationPikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks
Low-precision quantization is recognized for its efficacy in neural network optimization. Our analysis reveals that non-quantized elementwise operations which are prevalent in layers such as parameterized activation …
QuantizationPTQ-SL: Exploring the Sub-layerwise Post-training Quantization
Network quantization is a powerful technique to compress convolutional neural networks. The quantization granularity determines how to share the scaling factors in weights, which affects the performance of network quanti…
QuantizationA Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness
Neural networks are getting better accuracy with higher energy and computational cost. After quantization, the cost can be greatly saved, and the quantized models are more hardware friendly with acceptable accuracy loss.…
Quantization