ThriftyNets : Convolutional Neural Networks with Tiny Parameter Budget
Typical deep convolutional architectures present an increasing number of feature maps as we go deeper in the network, whereas spatial resolution of inputs is decreased through downsampling operations. This means that most of the parameters lay in the final layers, while a large portion of the computations are performed by a small fraction of the total parameters in the first layers. In an effort to use every parameter of a network at its maximum, we propose a new convolutional neural network architecture, called ThriftyNet. In ThriftyNet, only one convolutional layer is defined and used recursively, leading to a maximal parameter factorization. In complement, normalization, non-linearities, downsamplings and shortcut ensure sufficient expressivity of the model. ThriftyNet achieves competitive performance on a tiny parameters budget, exceeding 91% accuracy on CIFAR-10 with less than 40K parameters in total, and 74.3% on CIFAR-100 with less than 600K parameters.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Tiny-CRNN: Streaming Wakeword Detection In A Low Footprint Setting
In this work, we propose Tiny-CRNN (Tiny Convolutional Recurrent Neural Network) models applied to the problem of wakeword detection, and augment them with scaled dot product attention. We find that, compared to Convolut…
Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length
Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. TinyBERT addresses the computational efficiency by self-distilling BERT into a smaller …
Computational EfficiencyHyperparameter OptimizationQuestion AnsweringTinyCast: Probabilistic Zero-Shot Forecasting with Computed Periodicity
We introduce TinyCast, an attention-free zero-shot forecaster that emits a predictive distribution from 146,505 parameters, on the premise that at this size the periodic structure of a context is worth computing rather t…
msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML
AI spans from large language models to tiny models running on microcontrollers (MCUs). Extremely memory-efficient model architectures are decisive to fit within an MCU's tiny memory budget e.g., 128kB of RAM. However, in…
DenseScout: Algorithm-System Co-design for Budgeted Tiny Object Selection on Edge Platforms
Deploying high-resolution tiny-object perception on edge platforms requires not only accurate localization, but also selecting a small set of informative patches under compute, transport, and latency constraints. We stud…