Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems
Numerical Question Answering is the task of answering questions that require numerical capabilities. Previous works introduce general adversarial attacks to Numerical Question Answering, while not systematically exploring numerical capabilities specific to the topic. In this paper, we propose to conduct numerical capability diagnosis on a series of Numerical Question Answering systems and datasets. A series of numerical capabilities are highlighted, and corresponding dataset perturbations are designed. Empirical results indicate that existing systems are severely challenged by these perturbations. E.g., Graph2Tree experienced a 53.83% absolute accuracy drop against the `Extra'' perturbation on ASDiv-a, and BART experienced 13.80% accuracy drop against the `Language'' perturbation on the numerical subset of DROP. As a counteracting approach, we also investigate the effectiveness of applying perturbations as data augmentation to relieve systems' lack of robust numerical capabilities. With experiment analysis and empirical studies, it is demonstrated that Numerical Question Answering with robust numerical capabilities is still to a large extent an open question. We discuss future directions of Numerical Question Answering and summarize guidelines on future dataset collection and system design.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationOpen-Ended Question AnsweringQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering
Numerical reasoning over text is a challenging subtask in question answering (QA) that requires both the understanding of texts and numbers. However, existing language models in these numerical reasoning QA models tend t…
HallucinationQuestion AnsweringImproving Numerical Reasoning Skills in the Modular Approach for Complex Question Answering on Text
Numerical reasoning skills are essential for complex question answering (CQA) over text. It requires opertaions including counting, comparison, addition and subtraction. A successful approach to CQA on text, Neural Modul…
Question AnsweringTowards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks
Numerical reasoning is often important to accurately understand the world. Recently, several format-specific datasets have been proposed, such as numerical reasoning in the settings of Natural Language Inference (NLI), R…
Natural Language InferenceQuestion AnsweringReading ComprehensionA Pretraining Numerical Reasoning Model for Ordinal Constrained Question Answering on Knowledge Base
Knowledge Base Question Answering (KBQA) is to answer natural language questions posed over knowledge bases (KBs). This paper targets at empowering the IR-based KBQA models with the ability of numerical reasoning for ans…
Knowledge Base Question AnsweringQuestion AnsweringInjecting Numerical Reasoning Skills into Knowledge Base Question Answering Models
Embedding-based methods are popular for Knowledge Base Question Answering (KBQA), but few current models have numerical reasoning skills and thus struggle to answer ordinal constrained questions. This paper proposes a ne…
Data AugmentationKnowledge Base Question AnsweringQuestion Answering