AgentCompress: Task-Aware Compression for Affordable Large Language Model Agents
Large language models hold considerable promise for various applications, but their computational requirements create a barrier that many institutions cannot overcome. A single session using a 70-billion-parameter model can cost around $127 in cloud computing fees, which puts these tools out of reach for organizations operating on limited budgets. We present AgentCompress, a framework that tackles this problem through task-aware dynamic compression. The idea comes from a simple observation: not all tasks require the same computational effort. Complex reasoning, for example, is far more demanding than text reformatting, yet conventional compression applies the same reduction to both. Our approach uses a lightweight neural controller that looks at the first few tokens of each request, estimates how complex the task will be, and sends it to an appropriately quantized version of the model. This routing step adds only about 12 milliseconds of overhead. We tested the framework on 290 multi-stage workflows from domains including computer science, physics, chemistry, and biology. The results show a 68.3% reduction in computational costs while preserving 96.2% of the original success rate. These findings suggest that routing queries intelligently can make powerful language models substantially more affordable without sacrificing output quality
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Towards Efficient Agents: A Co-Design of Inference Architecture and System
The rapid development of large language model (LLM)-based agents has unlocked new possibilities for autonomous multi-turn reasoning and tool-augmented decision-making. However, their real-world deployment is hindered by …
A Decade of Research for Image Compression In Multimedia Laboratory
With the advancement of technology, we have supercomputers with high processing power and affordable prices. In addition, using multimedia expanded all around the world. This caused a vast use of images and videos in dif…
Data CompressionImage CompressionTensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition
High-dimensional token embeddings underpin Large Language Models (LLMs), as they can capture subtle semantic information and significantly enhance the modelling of complex language patterns. However, this high dimensiona…
Model CompressionBridging the Resource Gap: Deploying Advanced Imitation Learning Models onto Affordable Embedded Platforms
Advanced imitation learning with structures like the transformer is increasingly demonstrating its advantages in robotics. However, deploying these large-scale models on embedded platforms remains a major challenge. In t…
Imitation LearningModel CompressionSoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but the billion-scale parameters pose deployment challenges. Although existing methods attempt to reduce the scale of LLMs, the…
Model Compression