Context encoders as a simple but powerful extension of word2vec
With a simple architecture and the ability to learn meaningful word embeddings efficiently from texts containing billions of words, word2vec remains one of the most popular neural language models used today. However, as only a single embedding is learned for every word in the vocabulary, the model fails to optimally represent words with multiple meanings. Additionally, it is not possible to create embeddings for new (out-of-vocabulary) words on the spot. Based on an intuitive interpretation of the continuous bag-of-words (CBOW) word2vec model's negative sampling training objective in terms of predicting context based similarities, we motivate an extension of the model we call context encoders (ConEc). By multiplying the matrix of trained word2vec embeddings with a word's average context vector, out-of-vocabulary (OOV) embeddings and representations for a word with multiple meanings can be created based on the word's local contexts. The benefits of this approach are illustrated by using these word embeddings as features in the CoNLL 2003 named entity recognition (NER) task.
Code (1)
Tasks
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERWord EmbeddingsSimilar Papers 제목 키워드 기반
Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosody
Generating expressive and contextually appropriate prosody remains a challenge for modern text-to-speech (TTS) systems. This is particularly evident for long, multi-sentence inputs. In this paper, we examine simple exten…
Language ModelingLanguage ModellingSentencetext-to-speech+1Simple and Effective Graph Autoencoders with One-Hop Linear Models
Over the last few years, graph autoencoders (AE) and variational autoencoders (VAE) emerged as powerful node embedding methods, with promising performances on challenging tasks such as link prediction and node clustering…
ClusteringLink PredictionNode ClusteringKeep It Simple: Graph Autoencoders Without Graph Convolutional Networks
Graph autoencoders (AE) and variational autoencoders (VAE) recently emerged as powerful node embedding methods, with promising performances on challenging tasks such as link prediction and node clustering. Graph AE, VAE …
ClusteringLink PredictionNode ClusteringLLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich …
Contrastive LearningDecoderSimple Embedding-Based Word Sense Disambiguation
We present a simple knowledge-based WSD method that uses word and sense embeddings to compute the similarity between the gloss of a sense and the context of the word. Our method is inspired by the Lesk algorithm as it ex…
Word Sense Disambiguation