Design and Scheduling of an AI-based Queueing System
To leverage prediction models to make optimal scheduling decisions in service systems, we must understand how predictive errors impact congestion due to externalities on the delay of other jobs. Motivated by applications where prediction models interact with human servers (e.g., content moderation), we consider a large queueing system comprising of many single server queues where the class of a job is estimated using a prediction model. By characterizing the impact of mispredictions on congestion cost in heavy traffic, we design an index-based policy that incorporates the predicted class information in a near-optimal manner. Our theoretical results guide the design of predictive models by providing a simple model selection procedure with downstream queueing performance as a central concern, and offer novel insights on how to design queueing systems with AI-based triage. We illustrate our framework on a content moderation task based on real online comments, where we construct toxicity classifiers by finetuning large language models.
Code (0)
등록된 구현이 없습니다.
Tasks
Model SelectionPredictionSchedulingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Queueing, Predictions, and LLMs: Challenges and Open Problems
Queueing systems present many opportunities for applying machine-learning predictions, such as estimated service times, to improve system performance. This integration raises numerous open questions about how predictions…
Large Language ModelSchedulingThroughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents
As demand for Large Language Models (LLMs) and AI agents rapidly grows, optimizing systems for efficient LLM inference becomes critical. While significant efforts have focused on system-level engineering, little is explo…
AI AgentLarge Language ModelSchedulingScheduling Servers with Stochastic Bilinear Rewards
We address a control system optimization problem that arises in multi-class, multi-server queueing system scheduling with uncertainty. In this scenario, jobs incur holding costs while awaiting completion, and job-server …
SchedulingLearning-NUM: Network Utility Maximization with Unknown Utility Functions and Queueing Delay
Network Utility Maximization (NUM) studies the problems of allocating traffic rates to network users in order to maximize the users' total utility subject to network resource constraints. In this paper, we propose a new …
SchedulingUnbounded Human Learning: Optimal Scheduling for Spaced Repetition
In the study of human learning, there is broad evidence that our ability to retain information improves with repeated exposure and decays with delay since last exposure. This plays a crucial role in the design of educati…
Scheduling