DeepShare: Sharing ReLU Across Channels and Layers for Efficient Private Inference
Private Inference (PI) uses cryptographic primitives to perform privacy preserving machine learning. In this setting, the owner of the network runs inference on the data of the client without learning anything about the data and without revealing any information about the model. It has been observed that a major computational bottleneck of PI is the calculation of the gate (i.e., ReLU), so a considerable amount of effort have been devoted to reducing the number of ReLUs in a given network. We focus on the DReLU, which is the non-linear step function of the ReLU and show that one DReLU can serve many ReLU operations. We suggest a new activation module where the DReLU operation is only performed on a subset of the channels (Prototype channels), while the rest of the channels (replicate channels) replicates the DReLU of each of their neurons from the corresponding neurons in one of the prototype channels. We then extend this idea to work across different layers. We show that this formulation can drastically reduce the number of DReLU operations in resnet type network. Furthermore, our theoretical analysis shows that this new formulation can solve an extended version of the XOR problem, using just one non-linearity and two neurons, something that traditional formulations and some PI specific methods cannot achieve. We achieve new SOTA results on several classification setups, and achieve SOTA results on image segmentation.
Code (0)
등록된 구현이 없습니다.
Tasks
Image SegmentationSimilar Papers 제목 키워드 기반
Approximation Properties of Deep ReLU CNNs
This paper focuses on establishing $L^2$ approximation properties for deep ReLU convolutional neural networks (CNNs) in two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels wi…
Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning
Activation functions are essential for deep learning methods to learn and perform complex tasks such as image classification. Rectified Linear Unit (ReLU) has been widely used and become the default activation function a…
General Classificationimage-classificationImage ClassificationDrastically Reducing the Number of Trainable Parameters in Deep CNNs by Inter-layer Kernel-sharing
Deep convolutional neural networks (DCNNs) have become the state-of-the-art (SOTA) approach for many computer vision tasks: image classification, object detection, semantic segmentation, etc. However, most SOTA networks …
Edge-computingimage-classificationImage Classificationobject-detection+2Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinem…
Tied Block Convolution: Leaner and Better CNNs with Shared Thinner Filters
Convolution is the main building block of convolutional neural networks (CNN). We observe that an optimized CNN often has highly correlated filters as the number of channels increases with depth, reducing the expressive …
Instance Segmentationobject-detectionObject DetectionSemantic Segmentation