Semantic Vector Encoding and Similarity Search Using Fulltext Search Engines
Vector representations and vector space modeling (VSM) play a central role in modern machine learning. We propose a novel approach to {`}vector similarity searching{'} over dense semantic representations of words and documents that can be deployed on top of traditional inverted-index-based fulltext engines, taking advantage of their robustness, stability, scalability and ubiquity. We show that this approach allows the indexing and querying of dense vectors in text domains. This opens up exciting avenues for major efficiency gains, along with simpler deployment, scaling and monitoring. The end result is a fast and scalable vector database with a tunable trade-off between vector search performance and quality, backed by a standard fulltext engine such as Elasticsearch. We empirically demonstrate its querying performance and quality by applying this solution to the task of semantic searching over a dense vector representation of the entire English Wikipedia.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalRepresentation LearningSimilar Papers 제목 키워드 기반
Weight Initialization in Neural Language Models
Semantic Similarity is an important application which finds its use in many downstream NLP applications. Though the task is mathematically defined, semantic similarity's essence is to capture the notions of similarity im…
Semantic SimilaritySemantic Textual SimilarityVectorSearch: Enhancing Document Retrieval with Semantic Embeddings and Optimized Search
Traditional retrieval methods have been essential for assessing document similarity but struggle with capturing semantic nuances. Despite advancements in latent semantic analysis (LSA) and deep learning, achieving compre…
RetrievalKeyphrase Generation Beyond the Boundaries of Title and Abstract
Keyphrase generation aims at generating important phrases (keyphrases) that best describe a given document. In scholarly domains, current approaches have largely used only the title and abstract of the articles to genera…
ArticlesDecoderKeyphrase GenerationDifferential syntactic and semantic encoding in LLMs
We study how syntactic and semantic information is encoded in inner layer representations of Large Language Models (LLMs), focusing on the very large DeepSeek-V3. We find that, by averaging hidden-representation vectors …
Sparse Ternary Codes for similarity search have higher coding gain than dense binary codes
This paper addresses the problem of Approximate Nearest Neighbor (ANN) search in pattern recognition where feature vectors in a database are encoded as compact codes in order to speed-up the similarity search in large-sc…
Binarization