An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2
This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy demands with quantization due to lower throughput and some accuracy losses. Conversely, pruning reduces energy usage but impairs performance. The results highlight challenges and trade-offs in LLM model compression. We suggest future work on hardware-optimized quantization to enhance efficiency with minimal loss in accuracy.
Code (1)
Tasks
Model CompressionQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Energy Efficient Hardware Acceleration of Neural Networks with Power-of-Two Quantisation
Deep neural networks virtually dominate the domain of most modern vision systems, providing high performance at a cost of increased computational complexity.Since for those systems it is often required to operate both in…
Autonomous VehiclesOptimisation of a Siamese Neural Network for Real-Time Energy Efficient Object Tracking
In this paper the research on optimisation of visual object tracking using a Siamese neural network for embedded vision systems is presented. It was assumed that the solution shall operate in real-time, preferably for a …
Object TrackingVisual Object TrackingInference Energy and Latency in AI-Mediated Education: A Learning-per-Watt Analysis of Edge and Cloud Models
Immediate feedback is a foundational requirement of effective AI-mediated learning, yet the energy and latency costs of delivering it remain largely unexamined. This study investigates the latency-energy-learning trade-o…
Optimisation of the PointPillars network for 3D object detection in point clouds
In this paper we present our research on the optimisation of a deep neural network for 3D object detection in a point cloud. Techniques like quantisation and pruning available in the Brevitas and PyTorch tools were used.…
3D Object Detectionobject-detectionObject DetectionEnergy-Aware JPEG Image Compression: A Multi-Objective Approach
Customer satisfaction is crucially affected by energy consumption in mobile devices. One of the most energy-consuming parts of an application is images. While different images with different quality consume different amo…
Image Compression