SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers

2019-05-28 · NeurIPS 2019 12 · Igor Fedorov, Ryan P. Adams, Matthew Mattina, Paul N. Whatmough

The vast majority of processors in the world are actually microcontroller units (MCUs), which find widespread use performing simple control tasks in applications ranging from automobiles to medical devices and office equipment. The Internet of Things (IoT) promises to inject machine learning into many of these every-day objects via tiny, cheap MCUs. However, these resource-impoverished hardware platforms severely limit the complexity of machine learning models that can be deployed. For example, although convolutional neural networks (CNNs) achieve state-of-the-art results on many visual recognition tasks, CNN inference on MCUs is challenging due to severe finite memory limitations. To circumvent the memory challenge associated with CNNs, various alternatives have been proposed that do fit within the memory budget of an MCU, albeit at the cost of prediction accuracy. This paper challenges the idea that CNNs are not suitable for deployment on MCUs. We demonstrate that it is possible to automatically design CNNs which generalize well, while also being small enough to fit onto memory-limited MCUs. Our Sparse Architecture Search method combines neural architecture search with pruning in a single, unified approach, which learns superior models on four popular IoT datasets. The CNNs we find are more accurate and up to $4.35\times$ smaller than previous approaches, while meeting the strict MCU working memory constraint.

📄 PDF Abstract BibTeX arXiv:1905.12107

Code (0)

등록된 구현이 없습니다.

Tasks

BIG-bench Machine Learning Neural Architecture Search

Methods 이 논문이 사용한 방법론

Pruning 설명 없음

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

2019-10-02 · Souvik Kundu, Saurav Prakash, Haleh Akrami, Peter A. Beerel 외

The high demand for computational and storage resources severely impede the deployment of deep convolutional neural networks (CNNs) in limited-resource devices. Recent CNN architectures have proposed reduced complexity v…

SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training

2020-07-21 · Pengcheng Dai, Jianlei Yang, Xucheng Ye, Xingzhou Cheng 외

Training Convolutional Neural Networks (CNNs) usually requires a large number of computational resources. In this paper, \textit{SparseTrain} is proposed to accelerate CNN training by fully exploiting the sparsity. It ma…

Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks

2024-08-10 · Jianfei Li, Han Feng, Ding-Xuan Zhou

In this work, we explore intersections between sparse coding and deep learning to enhance our understanding of feature extraction capabilities in advanced neural network architectures. We begin by introducing a novel cla…

Deep Learning

Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers

2025-09-05 · Svetlana Pavlitska, Haixi Fan, Konstantin Ditschuneit, J. Marius Zöllner arxiv

Robustifying convolutional neural networks (CNNs) against adversarial attacks remains challenging and often requires resource-intensive countermeasures. We explore the use of sparse mixture-of-experts (MoE) layers to imp…

Phantom: A High-Performance Computational Core for Sparse Convolutional Neural Networks

2021-11-09 · Mahmood Azhar Qureshi, Arslan Munir

Sparse convolutional neural networks (CNNs) have gained significant traction over the past few years as sparse CNNs can drastically decrease the model size and computations, if exploited befittingly, as compared to their…

Vocal Bursts Intensity Prediction