WRPN & Apprentice: Methods for Training and Inference using Low-Precision Numerics
Today's high performance deep learning architectures involve large models with numerous parameters. Low precision numerics has emerged as a popular technique to reduce both the compute and memory requirements of these large models. However, lowering precision often leads to accuracy degradation. We describe three schemes whereby one can both train and do efficient inference using low precision numerics without hurting accuracy. Finally, we describe an efficient hardware accelerator that can take advantage of the proposed low precision numerics.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
WRPN: Wide Reduced-Precision Networks
For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint…
WRPN: Training and Inference using Wide Reduced-Precision Networks
For computer vision applications, prior works have shown the efficacy of reducing the numeric precision of model parameters (network weights) in deep neural networks but also that reducing the precision of activations hu…
QuantizationApprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous pa…
image-classificationImage ClassificationKnowledge DistillationModel Compression+2SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
Deep quantization of neural networks (below eight bits) offers significant promise in reducing their compute and storage cost. Albeit alluring, without special techniques for training and optimization, deep quantization …
QuantizationHierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards
While apprenticeship learning has shown promise for inducing effective pedagogical policies directly from student interactions in e-learning environments, most existing approaches rely on optimal or near-optimal expert d…