LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training
Transformers are widely used in NLP tasks. However, current approaches to leveraging transformers to understand language expose one weak spot: Number understanding. In some scenarios, numbers frequently occur, especially in semi-structured data like tables. But current approaches to rich-number tasks with transformer-based language models abandon or lose some of the numeracy information - e.g., breaking numbers into sub-word tokens - which leads to many number-related errors. In this paper, we propose the LUNA framework which improves the numerical reasoning and calculation capabilities of transformer-based language models. With the number plugin of NumTok and NumBed, LUNA represents each number as a whole to model input. With number pre-training, including regression loss and model distillation, LUNA bridges the gap between number and vocabulary embeddings. To the best of our knowledge, this is the first work that explicitly injects numeracy capability into language models using Number Plugins. Besides evaluating toy models on toy tasks, we evaluate LUNA on three large-scale transformer models (RoBERTa, BERT, TabBERT) over three different downstream tasks (TATQA, TabFact, CrediTrans), and observe the performances of language models are constantly improved by LUNA. The augmented models also improve the official baseline of TAT-QA (EM: 50.15 -> 59.58) and achieve SOTA performance on CrediTrans (F1 = 86.17).
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LUNA: A Framework for Language Understanding and Naturalness Assessment
The evaluation of Natural Language Generation (NLG) models has gained increased attention, urging the development of metrics that evaluate various aspects of generated text. LUNA addresses this challenge by introducing a…
nlg evaluationText GenerationEnhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation
Transformer and its variants are a powerful class of architectures for sequential recommendation, owing to their ability of capturing a user's dynamic interests from their past interactions. Despite their success, Transf…
Self-Supervised LearningSequential RecommendationSecure and Robust Communications for Cislunar Space Networks
There is no doubt that the Moon has become the center of interest for commercial and international actors. Over the past decade, the number of planned long-term missions has increased dramatically. This makes the establi…
LUNA: Linear Universal Neural Attention with Generalization Guarantees
Scaling attention faces a critical bottleneck: the $\mathcal{O}(n^2)$ quadratic computational cost of softmax attention, which limits its application in long-sequence domains. While linear attention mechanisms reduce thi…
Computational EfficiencyCERT: Contrastive Self-supervised Learning for Language Understanding
Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able …
Natural Language UnderstandingSelf-Supervised LearningSentenceTranslation