Word2Bits - Quantized Word Vectors
Word vectors require significant amounts of memory and storage, posing issues to resource limited devices like mobile phones and GPUs. We show that high quality quantized word vectors using 1-2 bits per parameter can be learned by introducing a quantization function into Word2Vec. We furthermore show that training with the quantization function acts as a regularizer. We train word vectors on English Wikipedia (2017) and evaluate them on standard word similarity and analogy tasks and on question answering (SQuAD). Our quantized word vectors not only take 8-16x less space than full precision (32 bit) word vectors but also outperform them on word similarity tasks and question answering.
Code (1)
Tasks
QuantizationQuestion AnsweringWord SimilaritySimilar Papers 제목 키워드 기반
Low-bit quantization and quantization-aware training for small-footprint keyword spotting
We investigate low-bit quantization to reduce computational cost of deep neural network (DNN) based keyword spotting (KWS). We propose approaches to further reduce quantization bits via integrating quantization into keyw…
Keyword SpottingQuantizationSmall-Footprint Keyword SpottingPersonalized word representations Carrying Personalized Semantics Learned from Social Network Posts
Distributed word representations have been shown to be very useful in various natural language processing (NLP) application tasks. These word vectors learned from huge corpora very often carry both semantic and syntactic…
SentenceSentence CompletionA Quantum-Like Approach to Word Sense Disambiguation
This paper presents a novel algorithm for Word Sense Disambiguation (WSD) based on Quantum Probability Theory. The Quantum WSD algorithm requires concepts representations as vectors in the complex domain and thus we have…
SentenceSentence EmbeddingsWord Sense DisambiguationTowards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks
We investigate segmenting and clustering speech into low-bitrate phone-like sequences without supervision. We specifically constrain pretrained self-supervised vector-quantized (VQ) neural networks so that blocks of cont…
ClusteringSegmentationParameter Free Hierarchical Graph-Based Clustering for Analyzing Continuous Word Embeddings
Word embeddings are high-dimensional vector representations of words and are thus difficult to interpret. In order to deal with this, we introduce an unsupervised parameter free method for creating a hierarchical graphic…
ClusteringDimensionality ReductionWord Embeddings