Benchmarking Test-Time Unsupervised Deep Neural Network Adaptation on Edge Devices
The prediction accuracy of the deep neural networks (DNNs) after deployment at the edge can suffer with time due to shifts in the distribution of the new data. To improve robustness of DNNs, they must be able to update themselves to enhance their prediction accuracy. This adaptation at the resource-constrained edge is challenging as: (i) new labeled data may not be present; (ii) adaptation needs to be on device as connections to cloud may not be available; and (iii) the process must not only be fast but also memory- and energy-efficient. Recently, lightweight prediction-time unsupervised DNN adaptation techniques have been introduced that improve prediction accuracy of the models for noisy data by re-tuning the batch normalization (BN) parameters. This paper, for the first time, performs a comprehensive measurement study of such techniques to quantify their performance and energy on various edge devices as well as find bottlenecks and propose optimization opportunities. In particular, this study considers CIFAR-10-C image classification dataset with corruptions, three robust DNNs (ResNeXt, Wide-ResNet, ResNet-18), two BN adaptation algorithms (one that updates normalization statistics and the other that also optimizes transformation parameters), and three edge devices (FPGA, Raspberry-Pi, and Nvidia Xavier NX). We find that the approach that only updates the normalization parameters with Wide-ResNet, running on Xavier GPU, to be overall effective in terms of balancing multiple cost metrics. However, the adaptation overhead can still be significant (around 213 ms). The results strongly motivate the need for algorithm-hardware co-design for efficient on-device DNN adaptation.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingGPUimage-classificationImage ClassificationPredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data
Unsupervised domain adaptation methods aim to generalize well on unlabeled test data that may have a different (shifted) distribution from the training data. Such methods are typically developed on image data, and their …
BenchmarkingDomain AdaptationTime SeriesTime Series Analysis+1Better Practices for Domain Adaptation
Distribution shifts are all too common in real-world applications of machine learning. Domain adaptation (DA) aims to address this by providing various frameworks for adapting models to the deployment data without using …
BenchmarkingDomain AdaptationSource-Free Domain AdaptationTest-time Adaptation+1Realistic Evaluation of Test-Time Adaptation Algorithms: Unsupervised Hyperparameter Selection
Test-Time Adaptation (TTA) has recently emerged as a promising strategy for tackling the problem of machine learning model robustness under distribution shifts by adapting the model during inference without access to any…
BenchmarkingModel SelectionTest-time AdaptationKnowledge Distillation for BERT Unsupervised Domain Adaptation
A pre-trained language model, BERT, has brought significant performance improvements across a range of natural language processing tasks. Since the model is trained on a large corpus of diverse topics, it shows robust pe…
Domain AdaptationGeneral ClassificationKnowledge DistillationLanguage Modeling+4A Systematic Evaluation of Domain Adaptation Algorithms On Time Series Data
Unsupervised domain adaptation methods aim to generalize well on unlabeled test data that may have a different (shifted) distribution from the training data. Such methods are typically developed on image data, and their …
BenchmarkingDomain AdaptationModel SelectionTime Series+2