Toward Metric Indexes for Incremental Insertion and Querying
In this work we explore the use of metric index structures, which accelerate nearest neighbor queries, in the scenario where we need to interleave insertions and queries during deployment. This use-case is inspired by a real-life need in malware analysis triage, and is surprisingly understudied. Existing literature tends to either focus on only final query efficiency, often does not support incremental insertion, or does not support arbitrary distance metrics. We modify and improve three algorithms to support our scenario of incremental insertion and querying with arbitrary metrics, and evaluate them on multiple datasets and distance metrics while varying the value of $k$ for the desired number of nearest neighbors. In doing so we determine that our improved Vantage-Point tree of Minimum-Variance performs best for this scenario.
Code (0)
등록된 구현이 없습니다.
Tasks
Malware AnalysisSimilar Papers 제목 키워드 기반
iRAG: Advancing RAG for Videos with an Incremental Approach
Retrieval-augmented generation (RAG) systems combine the strengths of language generation and information retrieval to power many real-world applications like chatbots. Use of RAG for understanding of videos is appealing…
Information RetrievalRAGRetrievalRetrieval-augmented Generation+1A Pluggable Learned Index Method via Sampling and Gap Insertion
Database indexes facilitate data retrieval and benefit broad applications in real-world systems. Recently, a new family of index, named learned index, is proposed to learn hidden yet useful data distribution and incorpor…
BIG-bench Machine LearningRetrievalEfficient Immediate-Access Dynamic Indexing
In a dynamic retrieval system, documents must be ingested as they arrive, and be immediately findable by queries. Our purpose in this paper is to describe an index structure and processing regime that accommodates that r…
RetrievalAutomated Database Indexing using Model-free Reinforcement Learning
Configuring databases for efficient querying is a complex task, often carried out by a database administrator. Solving the problem of building indexes that truly optimize database access requires a substantial amount of …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)iRangeGraph: Improvising Range-dedicated Graphs for Range-filtering Nearest Neighbor Search
Range-filtering approximate nearest neighbor (RFANN) search is attracting increasing attention in academia and industry. Given a set of data objects, each being a pair of a high-dimensional vector and a numeric value, an…