Incremental IVF Index Maintenance for Streaming Vector Search
The prevalence of vector similarity search in modern machine learning applications and the continuously changing nature of data processed by these applications necessitate efficient and effective index maintenance techniques for vector search indexes. Designed primarily for static workloads, existing vector search indexes degrade in search quality and performance as the underlying data is updated unless costly index reconstruction is performed. To address this, we introduce Ada-IVF, an incremental indexing methodology for Inverted File (IVF) indexes. Ada-IVF consists of 1) an adaptive maintenance policy that decides which index partitions are problematic for performance and should be repartitioned and 2) a local re-clustering mechanism that determines how to repartition them. Compared with state-of-the-art dynamic IVF index maintenance strategies, Ada-IVF achieves an average of 2x and up to 5x higher update throughput across a range of benchmark workloads.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
PLAID SHIRTTT for Large-Scale Streaming Dense Retrieval
PLAID, an efficient implementation of the ColBERT late interaction bi-encoder using pretrained language models for ranking, consistently achieves state-of-the-art performance in monolingual, cross-language, and multiling…
RetrievalSPI: Query-Depth-Adaptive Indexing for Streaming RAG in Vector Databases
Vector databases (VecDBs) are increasingly deployed in retrieval-augmented generation (RAG) pipelines where query processing and document ingestion occur concurrently. The index layer needs to provide low-latency search …
Natural QuestionsSPFresh: Incremental In-Place Update for Billion-Scale Vector Search
Approximate Nearest Neighbor Search (ANNS) is now widely used in various applications, ranging from information retrieval, question answering, and recommendation, to search for similar high-dimensional vectors. As the am…
Information RetrievalQuestion AnsweringFast Second-Order Online Kernel Learning through Incremental Matrix Sketching and Decomposition
Online Kernel Learning (OKL) has attracted considerable research interest due to its promising predictive performance in streaming environments. Second-order approaches are particularly appealing for OKL as they often of…
Recommendation SystemsProximity Graph Maintenance for Fast Online Nearest Neighbor Search
Approximate Nearest Neighbor (ANN) search is a fundamental technique for (e.g.,) the deployment of recommender systems. Recent studies bring proximity graph-based methods into practitioners' attention -- proximity graph-…
QuantizationRecommendation Systems