Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers
Numeracy is the ability to understand and work with numbers. It is a necessary skill for composing and understanding documents in clinical, scientific, and other technical domains. In this paper, we explore different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composition, and propose a novel neural architecture that uses a continuous probability density function to model numerals from an open vocabulary. Our evaluation on clinical and scientific datasets shows that using hierarchical models to distinguish numerals from words improves a perplexity metric on the subset of numerals by 2 and 4 orders of magnitude, respectively, over non-hierarchical models. A combination of strategies can further improve perplexity. Our continuous probability density function model reduces mean absolute percentage errors by 18% and 54% in comparison to the second best strategy for each dataset, respectively.
Code (1)
Tasks
Language ModellingSimilar Papers 제목 키워드 기반
Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Context
Physical measurements constitute a large portion of numbers in academic papers, engineering reports, and web tables. Current benchmarks fall short of properly evaluating numeracy of pretrained language models on measurem…
Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Context
Physical measurements constitute a large portion of numbers in academic papers, engineering reports, and web tables. Current benchmarks fall short of properly evaluating numeracy of pretrained language models on measurem…
Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model
The transformer-based pre-trained language models have been tremendously successful in most of the conventional NLP tasks. But they often struggle in those tasks where numerical understanding is required. Some possible r…
Transfer LearningNumeracy enhances the Literacy of Language Models
Specialized number representations in NLP have shown improvements on numerical reasoning tasks like arithmetic word problems and masked number prediction. But humans also use numeracy to make better sense of world concep…
PredictionSentenceArithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models
State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers. Recent work suggests two main reasons for thi…
Contrastive LearningReading ComprehensionTable-to-Text GenerationText Generation