How to represent a word and predict it, too: Improving tied architectures for language modelling
Recent state-of-the-art neural language models share the representations of words given by the input and output mappings. We propose a simple modification to these architectures that decouples the hidden state from the word embedding prediction. Our architecture leads to comparable or better results compared to previous tied models and models without tying, with a much smaller number of parameters. We also extend our proposal to word2vec models, showing that tying is appropriate for general word prediction tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModellingPredictionRepresentation LearningWord EmbeddingsSimilar Papers 제목 키워드 기반
Classifying Graphemes in English Words Through the Application of a Fuzzy Inference System
In Linguistics, a grapheme is a written unit of a writing system corresponding to a phonological sound. In Natural Language Processing tasks, written language is analysed through two different mediums, word analysis, and…
Which Evaluations Uncover Sense Representations that Actually Make Sense?
Text representations are critical for modern natural language processing. One form of text representation, sense-specific embeddings, reflect a word{'}s sense in a sentence better than single-prototype word embeddings ti…
Coherence EvaluationSentenceWord EmbeddingsAre Some Words Worth More than Others?
Current evaluation metrics for language modeling and generation rely heavily on the accuracy of predicted (or generated) words as compared to a reference ground truth. While important, token-level accuracy only captures …
Language ModelingLanguage ModellingPredictionNormXLogit: The Head-on-Top Never Lies
The Transformer architecture has emerged as the dominant choice for building large language models (LLMs). However, with new LLMs emerging on a frequent basis, it is important to consider the potential value of architect…
Word EmbeddingsProsody leaks into the memories of words
The average predictability (aka informativity) of a word in context has been shown to condition word duration (Seyfarth, 2014). All else being equal, words that tend to occur in more predictable environments are shorter …