Modeling Order in Neural Word Embeddings at Scale
Natural Language Processing (NLP) systems commonly leverage bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. The resulting word-level distributed representations often ignore morphological information, though character-level embeddings have proven valuable to NLP tasks. We propose a new neural language model incorporating both word order and character order in its embedding. The model produces several vector spaces with meaningful substructure, as evidenced by its performance of 85.8% on a recent word-analogy task, exceeding best published syntactic word-analogy scores by a 58% error margin. Furthermore, the model includes several parallel training methods, most notably allowing a skip-gram network with 160 billion parameters to be trained overnight on 3 multi-core CPUs, 14x larger than the previous largest neural network.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingWord EmbeddingsSimilar Papers 제목 키워드 기반
Encoding word order in complex embeddings
Sequential word order is important when processing text. Currently, neural networks (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of …
Language ModelingLanguage ModellingMachine TranslationPosition+4Multi-source Multi-view Transfer Learning in Neural Topic Modeling with Pretrained Topic and Word Embeddings
Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity problem in short text or small collect…
Information RetrievalRetrievalTransfer LearningWord EmbeddingsCWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling
Most existing topic models rely on bag-of-words (BOW) representation, which limits their ability to capture word order information and leads to challenges with out-of-vocabulary (OOV) words in new documents. Contextualiz…
Document ClassificationLanguage ModellingNatural Language UnderstandingNER+3Learning Composition Models for Phrase Embeddings
Lexical embeddings can serve as useful representations for words for a variety of NLP tasks, but learning embeddings for phrases can be challenging. While separate embeddings are learned for each word, this is infeasible…
Language ModelingLanguage ModellingSemantic SimilaritySemantic Textual Similarity+1A Hybrid Approach for Aspect-Based Sentiment Analysis Using Deep Contextual Word Embeddings and Hierarchical Attention
The Web has become the main platform where people express their opinions about entities of interest and their associated aspects. Aspect-Based Sentiment Analysis (ABSA) aims to automatically compute the sentiment towards…
Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)Sentiment AnalysisWord Embeddings