Subword-level Composition Functions for Learning Word Embeddings
Subword-level information is crucial for capturing the meaning and morphology of words, especially for out-of-vocabulary entries. We propose CNN- and RNN-based subword-level composition functions for learning word embeddings, and systematically compare them with popular word-level and subword-level models (Skip-Gram and FastText). Additionally, we propose a hybrid training scheme in which a pure subword-level model is trained jointly with a conventional word-level embedding model based on lookup-tables. This increases the fitness of all types of subword-level word embeddings; the word-level embeddings can be discarded after training, leaving only compact subword-level representation with much smaller data volume. We evaluate these embeddings on a set of intrinsic and extrinsic tasks, showing that subword-level models have advantage on tasks related to morphology and datasets with high OOV rate, and can be combined with other types of embeddings.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning Word EmbeddingsWord EmbeddingsSimilar Papers 제목 키워드 기반
A Systematic Study of Leveraging Subword Information for Learning Word Representations
The use of subword-level information (e.g., characters, character n-grams, morphemes) has become ubiquitous in modern word representation learning. Its importance is attested especially for morphologically rich languages…
Dependency ParsingEntity TypingRepresentation LearningSegmentation+1PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding
We look into the task of \emph{generalizing} word embeddings: given a set of pre-trained word vectors over a finite vocabulary, the goal is to predict embedding vectors for out-of-vocabulary words, \emph{without} extra c…
POSPOS TaggingWord EmbeddingsWord SimilarityRepresentation Learning for Unseen Words by Bridging Subwords to Semantic Networks
Pre-trained word embeddings are widely used in various fields. However, the coverage of pre-trained word embeddings only includes words that appeared in corpora where pre-trained embeddings are learned. It means that the…
Representation LearningWord EmbeddingsTreat the Word As a Whole or Look Inside? Subword Embeddings Model Language Change and Typology
We use a variant of word embedding model that incorporates subword information to characterize the degree of compositionality in lexical semantics. Our models reveal some interesting yet contrastive patterns of long-term…
Word-level Language Identification Using Subword Embeddings for Code-mixed Bangla-English Social Media Data
This paper reports work on building a word-level language identification (LID) model for code-mixed Bangla-English social media data using subword embeddings, with an ultimate goal of using this LID module as the first s…
Language IdentificationPOS