Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increase
In this article, we explore the challenges and evolution of two key technologies in the current field of AI: Vision Transformer model and Large Language Model (LLM). Vision Transformer captures global information by splitting images into small pieces and leveraging Transformer's multi-head attention mechanism, but its high reference count and compute overhead limit deployment on mobile devices. At the same time, the rapid development of LLM has revolutionized natural language processing, but it also faces huge deployment challenges. To address these issues, we investigate model pruning techniques, with a particular focus on how to reduce redundant parameters without losing accuracy to accommodate personalized data and resource-constrained environments. In this paper, a new layered pruning strategy is proposed to distinguish the personalized layer from the common layer by compressed sensing and random sampling, thus significantly reducing the model parameters. Our experimental results show that the introduced step buffering mechanism further improves the accuracy of the model after pruning, providing new directions and possibilities for the deployment of efficient and personalized AI models on mobile devices in the future.
Code (0)
등록된 구현이 없습니다.
Tasks
compressed sensingLanguage ModelingLanguage ModellingLarge Language ModelMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Homomorphic Parameter Compression for Distributed Deep Learning Training
Distributed training of deep neural networks has received significant research interest, and its major approaches include implementations on multiple GPUs and clusters. Parallelization can dramatically improve the effici…
Deep LearningAccelerating Distributed Deep Learning using Lossless Homomorphic Compression
As deep neural networks (DNNs) grow in complexity and size, the resultant increase in communication overhead during distributed training has become a significant bottleneck, challenging the scalability of distributed tra…
Computational EfficiencyCPUDeep LearningGPUInducing Individual Students' Learning Strategies through Homomorphic POMDPs
Optimizing students' learning strategies is a crucial component in intelligent tutoring systems. Previous research has demonstrated the effectiveness of devising personalized learning strategies for students by modelling…
From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Deep neural networks (DNNs) have been widely used in many artificial intelligence (AI) tasks. However, deploying them brings significant challenges due to the huge cost of memory, energy, and computation. To address thes…
Knowledge DistillationModel CompressionQuantizationSurveyPrivacy Preserving Machine Learning Model Personalization through Federated Personalized Learning
The widespread adoption of Artificial Intelligence (AI) has been driven by significant advances in intelligent system research. However, this progress has raised concerns about data privacy, leading to a growing awarenes…
Federated LearningPrivacy Preserving