Differentiable Architecture Compression

2020-01-01 · ICLR 2020 1 · Shashank Singh, Ashish Khetan, Zohar Karnin

In many learning situations, resources at inference time are significantly more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient at inference time. Given a resource-intensive base architecture, DARC utilizes the training data to learn which sub-components can be replaced by cheaper alternatives. The high-level technique can be applied to any neural architecture, and we report experiments on state-of-the-art convolutional neural networks for image classification. For a WideResNet with 97.2% accuracy on CIFAR-10, we improve single-sample inference speed by 2.28X and memory footprint by 5.64X, with no accuracy loss. For a ResNet with 79.15% Top-1 accuracy on ImageNet, we improve batch inference speed by 1.29X and memory footprint by 3.57X with 1% accuracy loss. We also give theoretical Rademacher complexity bounds in simplified cases, showing how DARC avoids over-fitting despite over-parameterization.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

image-classification Image Classification Model Compression

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Average Pooling 설명 없음

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Batch Normalization 설명 없음

Bottleneck Residual Block A Bottleneck Residual Block is a variant of the residual block that utilises 1x1 convolutions to create a bottleneck. The…

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Residual Block Residual Blocks are skip-connection blocks that learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. They were introduced…

DARC: Differentiable ARchitecture Compression

2019-05-20 · Shashank Singh, Ashish Khetan, Zohar Karnin

image-classification Image Classification Model Compression Neural Architecture Search

AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search

2020-01-13 · Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang 외

Large pre-trained language models such as BERT have shown their effectiveness in various natural language processing tasks. However, the huge parameter size makes them difficult to be deployed in real-time applications t…

Knowledge Distillation Neural Architecture Search

Learned JPEG Compression for DNN Vision

2026-06-15 · Kaixiang Zheng, Ahmed H. Salamah, Siyu Chen, En-Hui Yang arxiv

JPEG, a lossy image compression technique designed for human viewers, has maintained its dominance for decades. However, in the era of artificial intelligence (AI), a substantial portion of image data, often compressed b…

Image Compression

DKM: Differentiable K-Means Clustering Layer for Neural Network Compression

2021-08-28 · ICLR 2022 4 · Minsik Cho, Keivan A. Vahid, Saurabh Adya, Mohammad Rastegari

Deep neural network (DNN) model compression for efficient on-device inference is becoming increasingly important to reduce memory requirements and keep user data on-device. To this end, we propose a novel differentiable …

Clustering Model Compression Neural Network Compression

Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks

2026-04-25 · Terry Gou, Puneet Gupta arxiv

In this work, we developed and tested 3 techniques for vector quantization (VQ) based model weight compression. To mitigate codebook collapse and enable end-to-end training, we adopted cosine similarity-based assignment.…

Neural Architecture Search Model Compression