Tiny Neural Models for Seq2Seq

2021-08-07 · Arun Kandoor

Semantic parsing models with applications in task oriented dialog systems require efficient sequence to sequence (seq2seq) architectures to be run on-device. To this end, we propose a projection based encoder-decoder model referred to as pQRNN-MAtt. Studies based on projection methods were restricted to encoder-only models, and we believe this is the first study extending it to seq2seq architectures. The resulting quantized models are less than 3.5MB in size and are well suited for on-device latency critical applications. We show that on MTOP, a challenging multilingual semantic parsing dataset, the average model performance surpasses LSTM based seq2seq model that uses pre-trained embeddings despite being 85x smaller. Furthermore, the model can be an effective student for distilling large pre-trained models such as T5/BERT.

📄 PDF Abstract BibTeX arXiv:2108.03340

Code (2)

2024-MindSpore-1/Code7/tree/main/crnn_seq2seq_ocr mindspore

tensorflow/models/tree/master/research/seq_flow_lite tf

Tasks

Decoder Semantic Parsing

Methods 이 논문이 사용한 방법론

Tanh Activation 설명 없음

Sigmoid Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Seq2Seq Seq2Seq, or Sequence To Sequence, is a model used in sequence prediction tasks, such as language modelling and machine translation. The idea is to use one…

TinySV: Speaker Verification in TinyML with On-device Learning

2024-06-03 · Massimo Pavan, Gioele Mombelli, Francesco Sinacori, Manuel Roveri

TinyML is a novel area of machine learning that gained huge momentum in the last few years thanks to the ability to execute machine learning algorithms on tiny devices (such as Internet-of-Things or embedded systems). In…

Keyword Spotting Speaker Verification

TinySeg: Model Optimizing Framework for Image Segmentation on Tiny Embedded Systems

2024-05-03 · Byungchul Chae, Jiae Kim, Seonyeong Heo

Image segmentation is one of the major computer vision tasks, which is applicable in a variety of domains, such as autonomous navigation of an unmanned aerial vehicle. However, image segmentation cannot easily materializ…

Autonomous Navigation Image Segmentation Segmentation Semantic Segmentation

Tiny Machine Learning: Progress and Futures

2024-03-28 · Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang 외

Tiny Machine Learning (TinyML) is a new frontier of machine learning. By squeezing deep learning models into billions of IoT devices and microcontrollers (MCUs), we expand the scope of AI applications and enable ubiquito…

Deep Learning

TinyVQA: Compact Multimodal Deep Neural Network for Visual Question Answering on Resource-Constrained Devices

2024-04-04 · Hasib-Al Rashid, Argho Sarkar, Aryya Gangopadhyay, Maryam Rahnemoonfar 외

Traditional machine learning models often require powerful hardware, making them unsuitable for deployment on resource-limited devices. Tiny Machine Learning (tinyML) has emerged as a promising approach for running machi…

Quantization Question Answering Visual Question Answering Visual Question Answering (VQA)

TinyMetaFed: Efficient Federated Meta-Learning for TinyML

2023-07-13 · Haoyu Ren, Xue Li, Darko Anicic, Thomas A. Runkler

The field of Tiny Machine Learning (TinyML) has made substantial advancements in democratizing machine learning on low-footprint devices, such as microcontrollers. The prevalence of these miniature devices raises the que…

Computational Efficiency Few-Shot Learning Meta-Learning