DeServe: Towards Affordable Offline LLM Inference via Decentralization
The rapid growth of generative AI and its integration into everyday workflows have significantly increased the demand for large language model (LLM) inference services. While proprietary models remain popular, recent advancements in open-source LLMs have positioned them as strong contenders. However, deploying these models is often constrained by the high costs and limited availability of GPU resources. In response, this paper presents the design of a decentralized offline serving system for LLM inference. Utilizing idle GPU resources, our proposed system, DeServe, decentralizes access to LLMs at a lower cost. DeServe specifically addresses key challenges in optimizing serving throughput in high-latency network environments. Experiments demonstrate that DeServe achieves a 6.7x-12.6x improvement in throughput over existing serving system baselines in such conditions.
Code (0)
등록된 구현이 없습니다.
Tasks
GPULanguage ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
CascadeServe: Unlocking Model Cascades for Inference Serving
Machine learning (ML) models are increasingly deployed to production, calling for efficient inference serving systems. Efficient inference serving is complicated by two challenges: (i) ML models incur high computational …
modelSchedulingAn Information-Theoretic Analysis of The Cost of Decentralization for Learning and Inference Under Privacy Constraints
In vertical federated learning (FL), the features of a data sample are distributed across multiple agents. As such, inter-agent collaboration can be beneficial not only during the learning phase, as is the case for stand…
Federated LearningVertical Federated LearningDefining Decentralization: An Ontological Perspective
Decentralization as a concept in computer science has existed for over half a century. Despite its fundamental role across domains such as security, distributed computing, artificial intelligence, cloud infrastructures, …
Federated LearningEarly Human Capital Accumulation and Decentralization
Decentralization is a centerpiece in Cameroonian's government institutions' design. This chapter elaborates a simple hierarchy model for the analysis of the effects of power devolution. The model predicts overall positiv…
Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontend
Articulatory-to-acoustic (forward) mapping is a technique to predict speech using various articulatory acquisition techniques as input (e.g. ultrasound tongue imaging, MRI, lip video). The advantage of lip video is that …
Lip to Speech SynthesisSpeech Synthesis