BeLLMan: Controlling LLM Congestion
Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferencing latency inflation and poor user experience. Our first-cut controller, named beLLMan, enables the LLM infrastructure to actively and progressively signal the first-party LLM application to adjust the output length in response to changing system load. On a real testbed with H100 GPUs, beLLMan helps keep inferencing latency under control (upto 8X lower end-to-end latency) and reduces energy consumption by 25% (while serving 19% more requests) during periods of congestion for a summarization workload.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Intelligent City Traffic Management and Public Transportation System
Intelligent Transportation System in case of cities is controlling traffic congestion and regulating the traffic flow. This paper presents three modules that will help in managing city traffic issues and ultimately gives…
ManagementDifference of Convex Functions Programming for Reinforcement Learning
Large Markov Decision Processes (MDPs) are usually solved using Approximate Dynamic Programming (ADP) methods such as Approximate Value Iteration (AVI) or Approximate Policy Iteration (API). The main contribution of this…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)CoSLight: Co-optimizing Collaborator Selection and Decision-making to Enhance Traffic Signal Control
Effective multi-intersection collaboration is pivotal for reinforcement-learning-based traffic signal control to alleviate congestion. Existing work mainly chooses neighboring intersections as collaborators. However, qui…
Decision MakingTraffic Signal ControlDecongestion by Representation: Learning to Improve Economic Welfare in Marketplaces
Congestion is a common failure mode of markets, where consumers compete inefficiently on the same subset of goods (e.g., chasing the same small set of properties on a vacation rental platform). The typical economic story…
Representation LearningFederated Intelligence for Active Queue Management in Inter-Domain Congestion
Active Queue Management (AQM) has been considered as a paradigm for the complicated network management task of mitigating congestion by controlling buffer of network link queues. However, finding the right parameters for…
Federated LearningManagement