PARQ: Piecewise-Affine Regularized Quantization
We develop a principled method for quantization-aware training (QAT) of large-scale machine learning models. Specifically, we show that convex, piecewise-affine regularization (PAR) can effectively induce the model parameters to cluster towards discrete values. We minimize PAR-regularized loss functions using an aggregate proximal stochastic gradient method (AProx) and prove that it has last-iterate convergence. Our approach provides an interpretation of the straight-through estimator (STE), a widely used heuristic for QAT, as the asymptotic form of PARQ. We conduct experiments to demonstrate that PARQ obtains competitive performance on convolution- and transformer-based vision tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationSimilar Papers 제목 키워드 기반
Quantization through Piecewise-Affine Regularization: Optimization and Statistical Guarantees
Optimization problems over discrete or quantized variables are very challenging in general due to the combinatorial nature of their search space. Piecewise-affine regularization (PAR) provides a flexible modeling and com…
From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference
Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understan…
QuantizationA piecewise ellipsoidal reachable set estimation method for continuous bimodal piecewise affine systems
In this work, the issue of estimation of reachable sets in continuous bimodal piecewise affine systems is studied. A new method is proposed, in the framework of ellipsoidal bounding, using piecewise quadratic Lyapunov fu…
Fast Piecewise-Affine Motion Estimation Without Segmentation
Current algorithmic approaches for piecewise affine motion estimation are based on alternating motion segmentation and estimation. We propose a new method to estimate piecewise affine motion fields directly without inter…
Motion EstimationMotion SegmentationSegmentationDissecting Deep Neural Networks
In exchange for large quantities of data and processing power, deep neural networks have yielded models that provide state of the art predication capabilities in many fields. However, a lack of strong guarantees on their…