Word Embedding Binarization with Semantic Information Preservation

2020-12-01 · COLING 2020 8 · Samarth Navali, Praneet Sherki, Ramesh Inturi, Vanraj Vala

With growing applications of Machine Learning in daily lives Natural Language Processing (NLP) has emerged as a heavily researched area. Finding its applications in tasks ranging from simple Q/A chatbots to Fully fledged conversational AI, NLP models are vital. Word and Sentence embedding are one of the most common starting points of any NLP task. A word embedding represents a given word in a predefined vector-space while maintaining vector relations with similar or dis-similar entities. As such different pretrained embedding such as Word2Vec, GloVe, fasttext have been developed. These embedding generated on millions of words are however very large in terms of size. Having embedding with floating point precision also makes the downstream evaluation slow. In this paper we present a novel method to convert continuous embedding to its binary representation, thus reducing the overall size of the embedding while keeping the semantic and relational knowledge intact. This will facilitate an option of porting such big embedding onto devices where space is limited. We also present different approaches suitable for different downstream tasks based on the requirement of contextual and semantic information. Experiments have shown comparable result in downstream tasks with 7 to 15 times reduction in file size and about 5 {\%} change in evaluation parameters.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

BinarizationSentenceSentence EmbeddingSentence-Embedding

Methods 이 논문이 사용한 방법론

GloVe GloVe Embeddings are a type of word embedding that encode the co-occurrence probability ratio between two words as vector differences. GloVe uses a weighted least squares…

fastText fastText embeddings exploit subword information to construct word embeddings. Representations are learnt of character $n$-grams, and words represented as the sum of the…

Near-lossless Binarization of Word Embeddings

2018-03-24 · Julien Tissier, Christophe Gravier, Amaury Habrard

Word embeddings are commonly used as a starting point in many NLP models to achieve state-of-the-art performances. However, with a large vocabulary and many dimensions, these floating-point representations are expensive …

Binarization Semantic Similarity Semantic Textual Similarity Sentiment Analysis+3

Subword Tokenization Strategies for Kurdish Word Embeddings

2025-11-18 · Ali Salehi, Cassandra L. Jacobs arxiv

We investigate tokenization strategies for Kurdish word embeddings by comparing word-level, morpheme-based, and BPE approaches on morphological similarity preservation tasks. We develop a BiLSTM-CRF morphological segment…

Three-stage binarization of color document images based on discrete wavelet transform and generative adversarial networks

2022-11-29 · Rui-Yang Ju, Yu-Shian Lin, Yanlin Jin, Chih-Chia Chen 외

The efficient extraction of text information from the background in degraded color document images is an important challenge in the preservation of ancient manuscripts. The imperfect preservation of ancient manuscripts h…

Avg Binarization Image Enhancement Semantic Segmentation+1

Imparting Interpretability to Word Embeddings while Preserving Semantic Structure

2018-07-19 · Lutfi Kerem Senel, Ihsan Utlu, Furkan Şahinuç, Haldun M. Ozaktas 외

As an ubiquitous method in natural language processing, word embeddings are extensively employed to map semantic properties of words into a dense vector representation. They capture semantic and syntactic relations among…

Word Embeddings Word Similarity

Semantic Leakage from Image Embeddings

2026-01-30 · Yiyi Chen, Qiongkai Xu, Desmond Elliott, Qiongxiu Li 외 arxiv

Image embeddings are generally assumed to pose limited privacy risk. We challenge this assumption by formalizing semantic leakage as the ability to recover semantic structures from compressed image embeddings. Surprising…