RPR: Random Partition Relaxation for Training; Binary and Ternary Weight Neural Networks

2020-01-04 · Lukas Cavigelli, Luca Benini

We present Random Partition Relaxation (RPR), a method for strong quantization of neural networks weight to binary (+1/-1) and ternary (+1/0/-1) values. Starting from a pre-trained model, we quantize the weights and then relax random partitions of them to their continuous values for retraining before re-quantizing them and switching to another weight partition for further adaptation. We demonstrate binary and ternary-weight networks with accuracies beyond the state-of-the-art for GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using an SGD-based training method that can easily be integrated into existing frameworks.

📄 PDF Abstract BibTeX arXiv:2001.01091

Code (0)

등록된 구현이 없습니다.

Tasks

Quantization

Methods 이 논문이 사용한 방법론

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Average Pooling 설명 없음

Local Response Normalization Local Response Normalization is a normalization layer that implements the idea of lateral inhibition. Lateral inhibition is a concept in neurobiology that refers to the…

Auxiliary Classifier Auxiliary Classifiers are type of architectural component that seek to improve the convergence of very deep networks. They are classifier heads we attach to layers before the…

Inception Module An Inception Module is an image model block that aims to approximate an optimal local sparse structure in a CNN. Put simply, it allows for us to use multiple types of filter…

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Relaxations for inference in restricted Boltzmann machines

2013-12-21 · Sida I. Wang, Roy Frostig, Percy Liang, Christopher D. Manning

We propose a relaxation-based approximate inference algorithm that samples near-MAP configurations of a binary pairwise Markov random field. We experiment on MAP inference tasks in several restricted Boltzmann machines. …

Rainbow Differential Privacy

2022-02-08 · Ziqi Zhou, Onur Günlü, Rafael G. L. D'Oliveira, Muriel Médard 외

We extend a previous framework for designing differentially private (DP) mechanisms via randomized graph colorings that was restricted to binary functions, corresponding to colorings in a graph, to multi-valued functions…

Efficient semidefinite-programming-based inference for binary and multi-class MRFs

2020-12-04 · NeurIPS 2020 12 · Chirag Pabbaraju, Po-Wei Wang, J. Zico Kolter

Probabilistic inference in pairwise Markov Random Fields (MRFs), i.e. computing the partition function or computing a MAP estimate of the variables, is a foundational problem in probabilistic graphical models. Semidefini…

Achieving Exact Cluster Recovery Threshold via Semidefinite Programming

2014-11-24 · Bruce Hajek, Yihong Wu, Jiaming Xu

The binary symmetric stochastic block model deals with a random graph of $n$ vertices partitioned into two equal-sized clusters, such that each pair of vertices is connected independently with probability $p$ within clus…

Stochastic Block Model

Learning Binary Trees via Sparse Relaxation

2020-09-28 · Valentina Zantedeschi, Matt Kusner, Vlad Niculae

One of the most classical problems in machine learning is how to learn binary trees that split data into meaningful partitions. From classification/regression via decision trees to hierarchical clustering, binary trees a…

Clustering regression