BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures
Serving large language models (LLMs) to millions of users requires efficient resource allocation and parallelism strategies. It is a labor intensive trial-and-error process to find such a strategy. We present BestServe, a novel framework for ranking serving strategies by estimating goodput under various operating scenarios. Supporting both collocated and disaggregated architectures, BestServe leverages an inference simulator built on an adapted roofline model and CPU-GPU dispatch dynamics. Our framework determines the optimal strategy in minutes on a single standard CPU, eliminating the need for costly benchmarking, while achieving predictions within a $20\%$ error margin. It appeals to be practical for rapid deployment planning because of its lightweight design and strong extensibility.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingCPUGPUSimilar Papers 제목 키워드 기반
Revisiting SLO and Goodput Metrics in LLM Serving
Large language models (LLMs) have achieved remarkable performance and are widely deployed in various applications, while the serving of LLM inference has raised concerns about user experience and serving throughput. Acco…
Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Serving
Any-to-Any models are an emerging class of multimodal models that accept combinations of text and multimodal data as input and generate them as output, introducing heterogeneous computation paths and component scaling ch…
SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
Existing Large Language Model (LLM) serving systems prioritize maximum throughput. They often neglect Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT), which leads to su…
Language ModelingLanguage ModellingLarge Language ModelManagement+1Send Pilot or Data? Leveraging Age of Channel State Information for Throughput Maximization
In this paper, we study the optimal timing for pilot and data transmissions to maximize effective throughput, also known as goodput, over a wireless fading channel. The receiver utilizes the received pilot signal and its…
SchedulingAccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications
In this paper, we consider a mixed-prompt scenario for a large language model (LLM) inference serving system that supports diverse applications with both short prompts and long prompts and heterogeneous SLOs for iteratio…
ChunkingGPULarge Language Model