Online Cascade Learning for Efficient Inference over Streams
Large Language Models (LLMs) have a natural role in answering complex queries about data streams, but the high computational cost of LLM inference makes them infeasible in many such tasks. We propose online cascade learning, the first approach to address this challenge. The objective here is to learn a "cascade" of models, starting with lower-capacity models (such as logistic regression) and ending with a powerful LLM, along with a deferral policy that determines the model to be used on a given input. We formulate the task of learning cascades online as an imitation-learning problem, where smaller models are updated over time imitating the collected LLM demonstrations, and give a no-regret algorithm for the problem. Experimental results across four benchmarks show that our method parallels LLMs in accuracy while cutting down inference costs by as much as 90% with strong robustness against input distribution shifts, underscoring its efficacy and adaptability in stream processing.
Code (1)
Tasks
Imitation LearningSimilar Papers 제목 키워드 기반
CascadeServe: Unlocking Model Cascades for Inference Serving
Machine learning (ML) models are increasingly deployed to production, calling for efficient inference serving systems. Efficient inference serving is complicated by two challenges: (i) ML models incur high computational …
modelSchedulingA Circular Window-based Cascade Transformer for Online Action Detection
Online action detection aims at the accurate action prediction of the current frame based on long historical observations. Meanwhile, it demands real-time inference on online streaming videos. In this paper, we advocate …
Action DetectionAction SegmentationOnline Action DetectionOnline Estimation and Community Detection of Network Point Processes for Event Streams
A common goal in network modeling is to uncover the latent community structure present among nodes. For many real-world networks, the true connections consist of events arriving as streams, which are then aggregated to f…
Community DetectionOnline Community DetectionPoint ProcessesVariational InferenceFrom Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascades
Standard LLM cascades improve efficiency by deferring difficult queries from weak to strong models. However, these systems are typically static: when faced with repeated or semantically similar queries, they redundantly …
Knowledge DistillationContinual Learning with Optimal Transport based Mixture Model
Online Class Incremental learning (CIL) is a challenging setting in Continual Learning (CL), wherein data of new tasks arrive in incoming streams and online learning models need to handle incoming data streams without re…
class-incremental learningClass Incremental LearningContinual LearningIncremental Learning+1