Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics
The existing word representation methods mostly limit their information source to word co-occurrence statistics. In this paper, we introduce ngrams into four representation methods: SGNS, GloVe, PPMI matrix, and its SVD factorization. Comprehensive experiments are conducted on word analogy and similarity tasks. The results show that improved word representations are learned from ngram co-occurrence statistics. We also demonstrate that the trained ngram representations are useful in many aspects such as finding antonyms and collocations. Besides, a novel approach of building co-occurrence matrix is proposed to alleviate the hardware burdens brought by ngrams.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation
Recently, a new training oaxe loss has proven effective to ameliorate the effect of multimodality for non-autoregressive translation (NAT), which removes the penalty of word order errors in the standard cross-entropy los…
Machine TranslationSentenceTranslationLSTM Neural Network for Textual Ngrams
Cognitive neuroscience is the study of how the human brain functions on tasks like decision making, language, perception and reasoning. Deep learning is a class of machine learning algorithms that use neural networks. Th…
Decision MakingDeep LearningWolves at SemEval-2018 Task 10: Semantic Discrimination based on Knowledge and Association
This paper describes the system submitted to SemEval 2018 shared task 10 {`}Capturing Dicriminative Attributes{'}. We use a combination of knowledge-based and co-occurrence features to capture the semantic difference bet…
AttributeRelationSemantic Textual SimilarityWord SimilarityBeyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram
Current genomic foundation models (GFMs) rely on extensive neural computation to implicitly approximate conserved biological motifs from single-nucleotide inputs. We propose Gengram, a conditional memory module that intr…
Verifying Heaps' law using Google Books Ngram data
This article is devoted to the verification of the empirical Heaps law in European languages using Google Books Ngram corpus data. The connection between word distribution frequency and expected dependence of individual …
Text Generation