Discrete Wavelet Transform for Efficient Word Embeddings and Sentence Encoding
Wavelets have emerged as a cutting edge technology in a number of fields. Concrete results of their application in image and signal processing suggest that wavelets can be effectively applied to Natural Language Processing (NLP) tasks to capture a variety of linguistic properties. In this paper, we leverage the power of applying Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We evaluate, intrinsically and extrinsically, how wavelets can effectively be used to consolidate important information in a word vector while reducing its dimensionality. We demonstrate the effectiveness of using the DWT based embeddings, together with Discrete Cosine Transform (DCT), to compresses a sentence with a dense amount of information in a fixed size vector based on locally varying word features. We show the efficacy of the proposed paradigm on downstream applications yielding comparable and even superior (in some tasks) results to other state of the art spectral models.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceSentence EmbeddingsWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Combining Discrete Wavelet and Cosine Transforms for Efficient Sentence Embedding
Wavelets have emerged as a cutting edge technology in a number of fields. Concrete results of their application in Image and Signal processing suggest that wavelets can be effectively applied to Natural Language Processi…
Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform
Wavelet transforms, a powerful mathematical tool, have been widely used in different domains, including Signal and Image processing, to unravel intricate patterns, enhance data representation, and extract meaningful feat…
Semantic SimilarityDWTSumm: Discrete Wavelet Transform for Document Summarization
Summarizing long, domain-specific documents with large language models (LLMs) remains challenging due to context limitations, information loss, and hallucinations, particularly in clinical and legal settings. We propose …
Document SummarizationSemantic SimilarityDisentangling continuous and discrete linguistic signals in transformer-based sentence embeddings
Sentence and word embeddings encode structural and semantic information in a distributed manner. Part of the information encoded -- particularly lexical information -- can be seen as continuous, whereas other -- like str…
SentenceSentence EmbeddingsWord EmbeddingsEfficient Sentence Embedding using Discrete Cosine Transform
Vector averaging remains one of the most popular sentence embedding methods in spite of its obvious disregard for syntactic structure. While more complex sequential or convolutional networks potentially yield superior cl…
ClassificationGeneral ClassificationSentenceSentence Embedding+1