Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types. Although character-level language models offer a partial solution in that they can create word types not attested in the training corpus, they do not capture the "bursty" distribution of such words. In this paper, we augment a hierarchical LSTM language model that generates sequences of word tokens character by character with a caching mechanism that learns to reuse previously generated words. To validate our model we construct a new open-vocabulary language modeling corpus (the Multilingual Wikipedia Corpus, MWC) from comparable Wikipedia articles in 7 typologically diverse languages and demonstrate the effectiveness of our model across this range of languages.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesLanguage ModelingLanguage ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Taboo Wordnet
This paper describes the development of an online lexical resource to help detection systems regulate and curb the use of offensive words online. With the growing prevalence of social media platforms, many conversations …
Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model
We show how the spellings of known words can help us deal with unknown words in open-vocabulary NLP tasks. The method we propose can be used to extend any closed-vocabulary generative model, but in this paper we specific…
Language ModelingLanguage ModellingSentenceVocal Bursts Valence PredictionContext encoders as a simple but powerful extension of word2vec
With a simple architecture and the ability to learn meaningful word embeddings efficiently from texts containing billions of words, word2vec remains one of the most popular neural language models used today. However, as …
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER+1Using Morphological Knowledge in Open-Vocabulary Neural Language Models
Languages with productive morphology pose problems for language models that generate words from a fixed vocabulary. Although character-based models allow any possible word type to be generated, they are linguistically na…
Language ModelingLanguage ModellingMorphological DisambiguationQuery3D: LLM-Powered Open-Vocabulary Scene Segmentation with Language Embedded 3D Gaussian
This paper introduces a novel method for open-vocabulary 3D scene querying in autonomous driving by combining Language Embedded 3D Gaussians with Large Language Models (LLMs). We propose utilizing LLMs to generate both c…
Autonomous Drivingobject-detectionObject DetectionOpen Vocabulary Object Detection+2