Character and Subword-Based Word Representation for Neural Language Modeling Prediction
Most of neural language models use different kinds of embeddings for word prediction. While word embeddings can be associated to each word in the vocabulary or derived from characters as well as factored morphological decomposition, these word representations are mainly used to parametrize the input, i.e. the context of prediction. This work investigates the effect of using subword units (character and factored morphological decomposition) to build output representations for neural language modeling. We present a case study on Czech, a morphologically-rich language, experimenting with different input and output representations. When working with the full training vocabulary, despite unstable training, our experiments show that augmenting the output word representations with character-based embeddings can significantly improve the performance of the model. Moreover, reducing the size of the output look-up table, to let the character-based embeddings represent rare words, brings further improvement.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMachine TranslationSpeech RecognitionWord EmbeddingsSimilar Papers 제목 키워드 기반
Patterns versus Characters in Subword-aware Neural Language Modeling
Words in some natural languages can have a composite structure. Elements of this structure include the root (that could also be composite), prefixes and suffixes with which various nuances and relations to other words ca…
Language ModelingLanguage ModellingConfusion2vec 2.0: Enriching Ambiguous Spoken Language Representations with Subwords
Word vector representations enable machines to encode human language for spoken language understanding and processing. Confusion2vec, motivated from human speech production and perception, is a word vector representation…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Intent DetectionNatural Language Understanding+4Learning to Generate Word Representations using Subword Information
Distributed representations of words play a major role in the field of natural language processing by encoding semantic and syntactic information of words. However, most existing works on learning word representations ty…
ChunkingLanguage ModelingLanguage ModellingNamed Entity Recognition (NER)+4Character-based Neural Networks for Sentence Pair Modeling
Sentence pair modeling is critical for many NLP tasks, such as paraphrase identification, semantic textual similarity, and natural language inference. Most state-of-the-art neural models for these tasks rely on pretraine…
Language ModelingLanguage ModellingNatural Language InferenceParaphrase Identification+4Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens
Standard pretrained language models operate on sequences of subword tokens without direct access to the characters that compose each token’s string representation. We probe the embedding layer of pretrained language mode…
Language ModelingLanguage Modelling