Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification

2021-01-10 · Yao Qiang, Supriya Tumkur Suresh Kumar, Marco Brocanelli, Dongxiao Zhu

On-device Deep Neural Networks (DNNs) have recently gained more attention due to the increasing computing power of the mobile devices and the number of applications in Computer Vision (CV), Natural Language Processing (NLP), and Internet of Things (IoTs). Unfortunately, the existing efficient convolutional neural network (CNN) architectures designed for CV tasks are not directly applicable to NLP tasks and the tiny Recurrent Neural Network (RNN) architectures have been designed primarily for IoT applications. In NLP applications, although model compression has seen initial success in on-device text classification, there are at least three major challenges yet to be addressed: adversarial robustness, explainability, and personalization. Here we attempt to tackle these challenges by designing a new training scheme for model compression and adversarial robustness, including the optimization of an explainable feature mapping objective, a knowledge distillation objective, and an adversarially robustness objective. The resulting compressed model is personalized using on-device private training data via fine-tuning. We perform extensive experiments to compare our approach with both compact RNN (e.g., FastGRNN) and compressed RNN (e.g., PRADO) architectures in both natural and adversarial NLP test settings.

📄 PDF Abstract BibTeX arXiv:2101.05624

Code (0)

등록된 구현이 없습니다.

Tasks

Adversarial Robustness General Classification Knowledge Distillation Model Compression text-classification Text Classification

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Prototype-based Personalized Pruning

2021-03-25 · Jangho Kim, Simyung Chang, Sungrack Yun, Nojun Kwak

Nowadays, as edge devices such as smartphones become prevalent, there are increasing demands for personalized services. However, traditional personalization methods are not suitable for edge devices because retraining or…

Keyword Spotting Model Compression

Clustering-driven Memory Compression for On-device Large Language Models

2026-01-24 · Ondrej Bohdal, Pramit Saha, Umberto Michieli, Mete Ozay 외 arxiv

Large language models (LLMs) often rely on user-specific memories distilled from past interactions to enable personalized generation. A common practice is to concatenate these memories with the input prompt, but this app…

Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets

2020-03-02 · Thu Dinh, Bao Wang, Andrea L. Bertozzi, Stanley J. Osher

Deep neural nets (DNNs) compression is crucial for adaptation to mobile devices. Though many successful algorithms exist to compress naturally trained DNNs, developing efficient and stable compression algorithms for robu…

Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization

2025-09-10 · Kai Yi arxiv

Distributed and federated learning are essential paradigms for training models across decentralized data sources while preserving privacy, yet communication overhead remains a major bottleneck. This dissertation explores…

Federated Learning Model Compression

Personalized Federated Learning with Communication Compression

2022-09-12 · El Houcine Bergou, Konstantin Burlachenko, Aritra Dutta, Peter Richtárik

In contrast to training traditional machine learning (ML) models in data centers, federated learning (FL) trains ML models over local datasets contained on resource-constrained heterogeneous edge devices. Existing FL alg…

Federated Learning Personalized Federated Learning