Tiny Word Embeddings Using Globally Informed Reconstruction
We reduce the model size of pre-trained word embeddings by a factor of 200 while preserving its quality. Previous studies in this direction created a smaller word embedding model by reconstructing pre-trained word representations from those of subwords, which allows to store only a smaller number of subword embeddings in the memory. However, previous studies that train the reconstruction models using only target words cannot reduce the model size extremely while preserving its quality. Inspired by the observation of words with similar meanings having similar embeddings, our reconstruction training learns the global relationships among words, which can be employed in various models for word embedding reconstruction. Experimental results on word similarity benchmarks show that the proposed method improves the performance of the all subword-based reconstruction models.
Code (0)
등록된 구현이 없습니다.
Tasks
Word EmbeddingsWord SimilaritySimilar Papers 제목 키워드 기반
Query Expansion with Locally-Trained Word Embeddings
Continuous space word embeddings have received a great deal of attention in the natural language processing and machine learning communities for their ability to model term similarity and other relationships. We study th…
Ad-Hoc Information RetrievalBIG-bench Machine LearningInformation RetrievalRetrieval+1TINYCD: A (Not So) Deep Learning Model For Change Detection
In this paper, we present a lightweight and effective change detection model, called TinyCD. This model has been designed to be faster and smaller than current state-of-the-art change detection models due to industrial n…
Building change detection for remote sensing imagesChange DetectionChange detection for remote sensing imagesDeep LearningSubword-based Compact Reconstruction of Word Embeddings
The idea of subword-based word embeddings has been proposed in the literature, mainly for solving the out-of-vocabulary (OOV) word problem observed in standard word-based word embeddings. In this paper, we propose a meth…
Word EmbeddingsInterpretable Word Embeddings via Informative Priors
Word embeddings have demonstrated strong performance on NLP tasks. However, lack of interpretability and the unsupervised nature of word embeddings have limited their use within computational social science and digital h…
Word EmbeddingsDeconstructing word embedding algorithms
Word embeddings are reliable feature representations of words used to obtain high quality results for various NLP applications. Uncontextualized word embeddings are used in many NLP tasks today, especially in resource-li…
Word Embeddings