PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices
As neural networks (NN) are deployed across diverse sectors, their energy demand correspondingly grows. While several prior works have focused on reducing energy consumption during training, the continuous operation of ML-powered systems leads to significant energy use during inference. This paper investigates how the configuration of on-device hardware-elements such as GPU, memory, and CPU frequency, often neglected in prior studies, affects energy consumption for NN inference with regular fine-tuning. We propose PolyThrottle, a solution that optimizes configurations across individual hardware components using Constrained Bayesian Optimization in an energy-conserving manner. Our empirical evaluation uncovers novel facets of the energy-performance equilibrium showing that we can save up to 36 percent of energy for popular models. We also validate that PolyThrottle can quickly converge towards near-optimal settings while satisfying application constraints.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian OptimizationCPUEfficient Neural NetworkGPUSimilar Papers 제목 키워드 기반
Benchmarking Deep Learning Models for Object Detection on Edge Computing Devices
Modern applications, such as autonomous vehicles, require deploying deep learning algorithms on resource-constrained edge devices for real-time image and video processing. However, there is limited understanding of the e…
Autonomous VehiclesBenchmarkingEdge-computingobject-detection+2The Effects of Partitioning Strategies on Energy Consumption in Distributed CNN Inference at The Edge
Nowadays, many AI applications utilizing resource-constrained edge devices (e.g., small mobile robots, tiny IoT devices, etc.) require Convolutional Neural Network (CNN) inference on a distributed system at the edge due …
GCoDE: Efficient Device-Edge Co-Inference for GNNs via Architecture-Mapping Co-Search
Graph Neural Networks (GNNs) have emerged as the state-of-the-art graph learning method. However, achieving efficient GNN inference on edge devices poses significant challenges, limiting their application in real-world e…
Graph LearningDVFO: Learning-Based DVFS for Energy-Efficient Edge-Cloud Collaborative Inference
Due to limited resources on edge and different characteristics of deep neural network (DNN) models, it is a big challenge to optimize DNN inference performance in terms of energy consumption and end-to-end latency on edg…
Collaborative InferenceCPUDeep Reinforcement LearningGPULightweight Remote Sensing Scene Classification on Edge Devices via Knowledge Distillation and Early-exit
As the development of lightweight deep learning algorithms, various deep neural network (DNN) models have been proposed for the remote sensing scene classification (RSSC) application. However, it is still challenging for…
Knowledge DistillationScene Classification