PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units
To amortize cost, cloud vendors providing DNN acceleration as a service to end-users employ consolidation and virtualization to share the underlying resources among multiple DNN service requests. This paper makes a case for a "preemptible" neural processing unit (NPU) and a "predictive" multi-task scheduler to meet the latency demands of high-priority inference while maintaining high throughput. We evaluate both the mechanisms that enable NPUs to be preemptible and the policies that utilize them to meet scheduling objectives. We show that preemptive NPU multi-tasking can achieve an average 7.8x, 1.4x, and 4.8x improvement in latency, throughput, and SLA satisfaction, respectively.
Code (1)
Tasks
SchedulingSimilar Papers 제목 키워드 기반
Improving TAS Adaptability with a Variable Temperature Threshold
Thermal-Aware Scheduling (TAS) provides methods to manage the thermal dissipation of a computing chip during task execution. These methods aim to avoid issues such as accelerated aging of the device, premature failure an…
SchedulingGenetic Algorithm-based Routing and Scheduling for Wildfire Suppression using a Team of UAVs
This paper addresses early wildfire management using a team of UAVs for the mitigation of fires. The early detection and mitigation systems help in alleviating the destruction with reduced resource utilization. A Genetic…
DiversityManagementSchedulingRegulating CPU Temperature With Thermal-Aware Scheduling Using a Reduced Order Learning Thermal Model
Modern real-time systems utilize considerable amounts of power while executing computation-intensive tasks. The execution of these tasks leads to significant power dissipation and heating of the device. It therefore resu…
CPUSchedulingRenewable energy management in smart home environment via forecast embedded scheduling based on Recurrent Trend Predictive Neural Network
Smart home energy management systems help the distribution grid operate more efficiently and reliably, and enable effective penetration of distributed renewable energy sources. These systems rely on robust forecasting, o…
energy managementManagementSchedulingDynamic Measurement Scheduling for Adverse Event Forecasting using Deep RL
Current clinical practice to monitor patients' health follows either regular or heuristic-based lab test (e.g. blood test) scheduling. Such practice not only gives rise to redundant measurements accruing cost, but may ev…
Deep Reinforcement LearningReinforcement LearningReinforcement Learning (RL)Scheduling