Deploying a Steered Query Optimizer in Production at Microsoft
Modern analytical workloads are highly heterogeneous and massively complex, making generic query optimizers untenable for many customers and scenarios. As a result, it is important to specialize these optimizers to instances of the workloads. In this paper, we continue a recent line of work in steering a query optimizer towards better plans for a given workload, and make major strides in pushing previous research ideas to production deployment. Along the way we solve several operational challenges including, making steering actions more manageable, keeping the costs of steering within budget, and avoiding unexpected performance regressions in production. Our resulting system, QQ-advisor, essentially externalizes the query planner to a massive offline pipeline for better exploration and specialization. We discuss various aspects of our design and show detailed results over production SCOPE workloads at Microsoft, where the system is currently enabled by default.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Web-Scale Generic Object Detection at Microsoft Bing
In this paper, we present Generic Object Detection (GenOD), one of the largest object detection systems deployed to a web-scale general visual search engine that can detect over 900 categories for all Microsoft Bing Visu…
Objectobject-detectionObject DetectionPhoebe: A Learning-based Checkpoint Optimizer
Easy-to-use programming interfaces paired with cloud-scale processing engines have enabled big data system users to author arbitrarily complex analytical jobs over massive volumes of data. However, as the complexity and …
Neuro-Symbolic Query Optimization in Knowledge Graphs
This chapter delves into the emerging field of neuro-symbolic query optimization for knowledge graphs (KGs), presenting a comprehensive exploration of how neural and symbolic techniques can be integrated to enhance query…
Knowledge GraphsNavigateRELOAD: A Robust and Efficient Learned Query Optimizer for Database Systems
Recent advances in query optimization have shifted from traditional rule-based and cost-based techniques towards machine learning-driven approaches. Among these, reinforcement learning (RL) has attracted significant atte…
Reinforcement LearningUnified Generative & Dense Retrieval for Query Rewriting in Sponsored Search
Sponsored search is a key revenue source for search engines, where advertisers bid on keywords to target users or search queries of interest. However, finding relevant keywords for a given query is challenging due to the…
RetrievalText GenerationText RetrievalUnity