Clustering-driven Memory Compression for On-device Large Language Models
Large language models (LLMs) often rely on user-specific memories distilled from past interactions to enable personalized generation. A common practice is to concatenate these memories with the input prompt, but this approach quickly exhausts the limited context available in on-device LLMs. Compressing memories by averaging can mitigate context growth, yet it frequently harms performance due to semantic conflicts across heterogeneous memories. In this work, we introduce a clustering-based memory compression strategy that balances context efficiency and personalization quality. Our method groups memories by similarity and merges them within clusters prior to concatenation, thereby preserving coherence while reducing redundancy. Experiments demonstrate that our approach substantially lowers the number of memory tokens while outperforming baseline strategies such as naive averaging or direct concatenation. Furthermore, for a fixed context budget, clustering-driven merging yields more compact memory representations and consistently enhances generation quality.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models
Since Large Language Models or LLMs have demonstrated high-quality performance on many complex language tasks, there is a great interest in bringing these LLMs to mobile devices for faster responses and better privacy pr…
ClusteringCPUQuantizationDKM: Differentiable K-Means Clustering Layer for Neural Network Compression
Deep neural network (DNN) model compression for efficient on-device inference is becoming increasingly important to reduce memory requirements and keep user data on-device. To this end, we propose a novel differentiable …
ClusteringModel CompressionNeural Network CompressionSCSP: Spectral Clustering Filter Pruning with Soft Self-adaption Manners
Deep Convolutional Neural Networks (CNN) has achieved significant success in computer vision field. However, the high computational cost of the deep complex models prevents the deployment on edge devices with limited mem…
ClusteringModel CompressionData-driven Clustering and Merging of Adapters for On-device Large Language Models
On-device large language models commonly employ task-specific adapters (e.g., LoRAs) to deliver strong performance on downstream tasks. While storing all available adapters is impractical due to memory constraints, mobil…
Memory Efficient Corner Detection for Event-driven Dynamic Vision Sensors
Event cameras offer low-latency and data compression for visual applications, through event-driven operation, that can be exploited for edge processing in tiny autonomous agents. Robust, accurate and low latency extracti…
Data CompressionEdge-computing