Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

2026-03-19 · Yifan Sui, Han Zhao, Rui Ma, Zhiyuan He, Hao Wang, Jianxun Li, Kaiqiang Xu, Kai Chen, Yuqing Yang arxiv

LLM-powered agents execute tasks through a sequential loop of model generation and tool execution. Today's serving systems serialize this loop, leaving tool latency exposed on the task critical path. This paper presents PASTE, a tool-aware agent-serving system that predicts concrete future tool invocations from recurring agent patterns and executes them speculatively while the LLM is still generating. PASTE isolates speculative results until confirmed by the LLM and jointly schedules tool execution and returning LLM sessions to avoid shifting bottlenecks to the GPU. Across deep research, coding, and scientific-agent workloads, PASTE reduces average task completion time by 43.5% and lowers observed tool latency by 1.8x.

📄 PDF Abstract BibTeX arXiv:2603.18897

Code (0)

등록된 구현이 없습니다.

AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

2026-08-01 · Hao Mark Chen, Jinnan Guo, Wayne Luk, Hongxiang Fan arxiv

Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step. As decoding accelerates, tool execution becomes a growing bottleneck. E…

Beyond State Machines: Executing Network Procedures with Agentic Tool-Calling Sequences

2026-05-04 · Purna Sai Garigipati, Onur Ayan, Kishor Chandra Joshi, Xueli An arxiv

Agentic AI will be an essential enabling technology for designing future mobile communication systems, which could provide flexible and customized services, automate complex network operations, and drive autonomous decis…

Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

2026-07-09 · Kalle Kujanpää, Ning Liu, Shahnawaz Alam, Yeshwanth Reddy Sura 외 arxiv

Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request. We replace this inference-time coding loop with an agentic tool-making pipeline that compiles…

Code Generation

Jenius Agent: Towards Experience-Driven Accuracy Optimization in Real-World Scenarios

2026-01-05 · Defei Xia, Bingfeng Pi, Shenbin Zhang, Song Hua 외 arxiv

As agent systems powered by large language models (LLMs) advance, improving performance in context understanding, tool usage, and long-horizon execution has become critical. However, existing agent frameworks and benchma…

MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems

2026-04-14 · Yifei Wang, Hancheng Ye, Yechen Xu, Cong Guo 외 arxiv

Large language models (LLMs) are increasingly deployed as the execution core of autonomous agents rather than as standalone text generators. Agentic workloads induce a temporal shift from single-turn inference to multi-t…