Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model
We show how the spellings of known words can help us deal with unknown words in open-vocabulary NLP tasks. The method we propose can be used to extend any closed-vocabulary generative model, but in this paper we specifically consider the case of neural language modeling. Our Bayesian generative story combines a standard RNN language model (generating the word tokens in each sentence) with an RNN-based spelling model (generating the letters in each word type). These two RNNs respectively capture sentence structure and word structure, and are kept separate as in linguistics. By invoking the second RNN to generate spellings for novel words in context, we obtain an open-vocabulary language model. For known words, embeddings are naturally inferred by combining evidence from type spelling and token context. Comparing to baselines (including a novel strong baseline), we beat previous work and establish state-of-the-art results on multiple datasets.
Code (1)
Tasks
Language ModelingLanguage ModellingSentenceVocal Bursts Valence PredictionSimilar Papers 제목 키워드 기반
AlayaWorld: Long-Horizon and Playable Video World Generation
Game worlds have traditionally been built through labor-intensive production pipelines, making them costly to develop, difficult to customization, and expensive to modify after deployment. Recent advances in video world …
Gaze Target Estimation Anywhere with Concepts
Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and hu…
Gaze Target EstimationGaze EstimationCoherent Wave Dynamics and Language Generation of a Generative Pre-trained Transformer
Large Language Models (LLMs), such as the Generative Pretrained Transformer (GPT), have achieved tremendous success in various language tasks, but their emergent abilities have also raised many questions, concerns, and c…
Text GenerationTiSpell: A Semi-Masked Methodology for Tibetan Spelling Correction covering Multi-Level Error with Data Augmentation
Multi-level Tibetan spelling correction addresses errors at both the character and syllable levels within a unified model. Existing methods focus mainly on single-level correction and lack effective integration of both l…
Data AugmentationSpelling CorrectionFrom Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes
3D visual grounding has made notable progress in localizing objects within complex 3D scenes. However, grounding referring expressions beyond objects in 3D scenes remains unexplored. In this paper, we introduce Anywhere3…
3D visual groundingObjectReferring ExpressionSpatial Reasoning+1