Robustness to distribution shifts of compressed networks for edge devices
It is necessary to develop efficient DNNs deployed on edge devices with limited computation resources. However, the compressed networks often execute new tasks in the target domain, which is different from the source domain where the original network is trained. It is important to investigate the robustness of compressed networks in two types of data distribution shifts: domain shifts and adversarial perturbations. In this study, we discover that compressed models are less robust to distribution shifts than their original networks. Interestingly, larger networks are more vulnerable to losing robustness than smaller ones, even when they are compressed to a similar size as the smaller networks. Furthermore, compact networks obtained by knowledge distillation are much more robust to distribution shifts than pruned networks. Finally, post-training quantization is a reliable method for achieving significant robustness to distribution shifts, and it outperforms both pruned and distilled models in terms of robustness.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Distributed Monitoring for Data Distribution Shifts in Edge-ML Fraud Detection
The digital era has seen a marked increase in financial fraud. edge ML emerged as a promising solution for smartphone payment services fraud detection, enabling the deployment of ML models directly on edge devices. This …
Fraud DetectionAttacking Compressed Vision Transformers
Vision Transformers are increasingly embedded in industrial systems due to their superior performance, but their memory and power requirements make deploying them to edge devices a challenging task. Hence, model compress…
Model CompressionEdge Radar Material Classification Under Geometry Shifts
Material awareness can improve robotic navigation and interaction, particularly in conditions where cameras and LiDAR degrade. We present a lightweight mmWave radar material classification pipeline designed for ultra-low…
Non-adversarial Robustness of Deep Learning Methods for Computer Vision
Non-adversarial robustness, also known as natural robustness, is a property of deep learning models that enables them to maintain performance even when faced with distribution shifts caused by natural variations in data.…
Adversarial RobustnessDeep LearningNon-Adversarial RobustnessKroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
The development of over-parameterized pre-trained language models has made a significant contribution toward the success of natural language processing. While over-parameterization of these models is the key to their gen…
Knowledge DistillationLanguage ModelingLanguage ModellingModel Compression