Corrected CBOW Performs as well as Skip-gram
Mikolov et al. (2013a) observed that continuous bag-of-words (CBOW) word embeddings tend to underperform Skip-gram (SG) embeddings, and this finding has been reported in subsequent works. We find that these observations are driven not by fundamental differences in their training objectives, but more likely on faulty negative sampling CBOW implementations in popular libraries such as the official implementation, word2vec.c, and Gensim. We show that after correcting a bug in the CBOW gradient update, one can learn CBOW word embeddings that are fully competitive with SG on various intrinsic and extrinsic tasks, while being many times faster to train.
Code (1)
Tasks
Word EmbeddingsSimilar Papers 제목 키워드 기반
基于强负采样的词嵌入优化算法(Word Embedding Optimization Based on Hard Negative Sampling)
word2vec是自然语言处理领域重要的词嵌入算法之一,为了解决随机负采样作为优化目标可能出现的样本贡献消失问题,提出了可以应用在CBOW和Skip-gram框架上的以余弦距离为度量的强负采样方法:HNS-CBOW和HNS-SG。将原随机负采样过程拆解为两个步骤,首先,计算随机负样本与目标词的余弦距离,然后,再使用距离较近的强负样本更新参数。以英文维基百科数据作为实验语料,在公开的语义-语法数据集上对优化算法的效果进行了定量分析,实验表…
Apport de l'information temporelle des contextes pour la repr\'esentation vectorielle continue des mots
Les repr{\'e}sentations vectorielles continues des mots sont en plein essor et ont d{\'e}j{\`a} {\'e}t{\'e} appliqu{\'e}es avec succ{\`e}s {\`a} de nombreuses t{\^a}ches en traitement automatique de la langue (TAL). Dans…
es-enLearning the Dimensionality of Word Embeddings
We describe a method for learning word embeddings with data-dependent dimensionality. Our Stochastic Dimensionality Skip-Gram (SD-SG) and Stochastic Dimensionality Continuous Bag-of-Words (SD-CBOW) are nonparametric anal…
Learning Word EmbeddingsWord EmbeddingsLearning Word Embedding with Better Distance Weighting and Window Size Scheduling
Distributed word representation (a.k.a. word embedding) is a key focus in natural language processing (NLP). As a highly successful word embedding model, Word2Vec offers an efficient method for learning distributed word …
SchedulingLearning to retrieve out-of-vocabulary words in speech recognition
Many Proper Names (PNs) are Out-Of-Vocabulary (OOV) words for speech recognition systems used to process diachronic audio data. To help recovery of the PNs missed by the system, relevant OOV PNs can be retrieved out of t…
Retrievalspeech-recognitionSpeech Recognition