Implementation Notes for the Soft Cosine Measure
The standard bag-of-words vector space model (VSM) is efficient, and ubiquitous in information retrieval, but it underestimates the similarity of documents with the same meaning, but different terminology. To overcome this limitation, Sidorov et al. proposed the Soft Cosine Measure (SCM) that incorporates term similarity relations. Charlet and Damnati showed that the SCM is highly effective in question answering (QA) systems. However, the orthonormalization algorithm proposed by Sidorov et al. has an impractical time complexity of $\mathcal O(n^4)$, where n is the size of the vocabulary. In this paper, we prove a tighter lower worst-case time complexity bound of $\mathcal O(n^3)$. We also present an algorithm for computing the similarity between documents and we show that its worst-case time complexity is $\mathcal O(1)$ given realistic conditions. Lastly, we describe implementation in general-purpose vector databases such as Annoy, and Faiss and in the inverted indices of text search engines such as Apache Lucene, and ElasticSearch. Our results enable the deployment of the SCM in real-world information retrieval systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalQuestion AnsweringRetrievalSimilar Papers 제목 키워드 기반
Outlier Detection Using Vector Cosine Similarity by Adding a Dimension
We propose a new outlier detection method for multi-dimensional data. The method detects outliers based on vector cosine similarity, using a new dataset constructed by adding a dimension with zero values to the original …
Outlier DetectionWhat Cosine Similarity of Label Representations Can and Cannot Tell us
Cosine similarity is often used to measure the similarity of vector representations of neural network models. However, the cosine similarity of representations is not guaranteed to tell us anything about model probabilit…
Hyperspectral holography and spectroscopy: computational features of inverse discrete cosine transform
Broadband hyperspectral digital holography and Fourier transform spectroscopy are important instruments in various science and application fields. In the digital hyperspectral holography and spectroscopy the variable of …
A Comparison of Semantic Similarity Methods for Maximum Human Interpretability
The inclusion of semantic information in any similarity measures improves the efficiency of the similarity measure and provides human interpretable results for further analysis. The similarity calculation method that foc…
Information RetrievalRetrievalSemantic SimilaritySemantic Textual SimilarityCosineGate: Semantic Dynamic Routing via Cosine Incompatibility in Residual Networks
Modern deep residual networks perform substantial redundant computation by evaluating all residual blocks for every input, even when identity mappings suffice. We introduce CosineGate, an end-to-end differentiable archit…