Work in Progress: Linear Transformers for TinyML
We present the WaveFormer, a neural network architecture based on a linear attention transformer to enable long sequence inference for TinyML devices. Waveformer achieves a new state-of-the-art accuracy of 98.8 % and 99.1 % on the Google Speech V2 keyword spotting (KWS) dataset for the 12 and 35 class problems with only 130 kB of weight storage, compatible with MCU class devices. Top-1 accuracy is improved by 0.1 and 0.9 percentage points while reducing the model size and number of operations by 2.5× and 4.7× compared to the state of the art. We also propose a hardware-friendly 8-bit integer quantization algorithm for the linear attention operator, enabling efficient deployment on low-cost, ultra-low-power microcontrollers without loss of accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Keyword SpottingKeyword Spotting on Google Speech CommandsQuantizationSimilar Papers 제목 키워드 기반
Tiny Machine Learning: Progress and Futures
Tiny Machine Learning (TinyML) is a new frontier of machine learning. By squeezing deep learning models into billions of IoT devices and microcontrollers (MCUs), we expand the scope of AI applications and enable ubiquito…
Deep LearningToward Attention-based TinyML: A Heterogeneous Accelerated Architecture and Automated Deployment Flow
One of the challenges for Tiny Machine Learning (tinyML) is keeping up with the evolution of Machine Learning models from Convolutional Neural Networks to Transformers. We address this by leveraging a heterogeneous archi…
Benchmarking TinyML Systems: Challenges and Direction
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark …
BenchmarkingPositionOn The Dynamic Ensemble Selection for TinyML-based Systems -- a Preliminary Study
The recent progress in TinyML technologies triggers the need to address the challenge of balancing inference time and classification quality. TinyML systems are defined by specific constraints in computation, memory and …
RedMule: A Mixed-Precision Matrix-Matrix Operation Engine for Flexible and Energy-Efficient On-Chip Linear Algebra and TinyML Training Acceleration
The increasing interest in TinyML, i.e., near-sensor machine learning on power budgets of a few tens of mW, is currently pushing toward enabling TinyML-class training as opposed to inference only. Current training algori…