FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices
Deploying LLMs on edge devices presents serious technical challenges. Memory elasticity is crucial for edge devices with unified memory, where memory is shared and fluctuates dynamically. Existing solutions suffer from either poor transition granularity or high storage costs. We propose FlexQuant, a novel elasticity framework that generates an ensemble of quantized models, providing an elastic hosting solution with 15x granularity improvement and 10x storage reduction compared to SoTA methods. FlexQuant works with most quantization methods and creates a family of trade-off options under various storage limits through our pruning method. It brings great performance and flexibility to the edge deployment of LLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Elastic Product Quantization for Time Series
Analyzing numerous or long time series is difficult in practice due to the high storage costs and computational requirements. Therefore, techniques have been proposed to generate compact similarity-preserving representat…
QuantizationTime SeriesTime Series AnalysisEQ-Net: Elastic Quantization Neural Networks
Current model quantization methods have shown their promising capability in reducing storage space and computation complexity. However, due to the diversity of quantization forms supported by different hardware, one limi…
QuantizationMinimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments
Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or r…
Validation of a Small Language Model for DSM-5 Substance Category Classification in Child Welfare Records
Background: Recent studies have demonstrated that large language models (LLMs) can perform binary classification tasks on child welfare narratives, detecting the presence or absence of constructs such as substance-relate…
Binary ClassificationQuEPT: Quantized Elastic Precision Transformers with One-Shot Calibration for Multi-Bit Switching
Elastic precision quantization enables multi-bit deployment via a single optimization pass, fitting diverse quantization scenarios.Yet, the high storage and optimization costs associated with the Transformer architecture…