Machine Reading, Fast and Slow: When Do Models “Understand” Language?
Two of the most fundamental issues in Natural Language Understanding (NLU) at present are: (a) how it can established whether deep learning-based models score highly on NLU benchmarks for the ”right” reasons; and (b) what those reasons would even be. We investigate the behavior of reading comprehension models with respect to two linguistic ”skills”: coreference resolution and comparison. We propose a definition for the reasoning steps expected from a system that would be ”reading slowly”, and compare that with the behavior of five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. We find that for comparison (but not coreference) the systems based on larger encoders are more likely to rely on the ”right” information, but even they struggle with generalization, suggesting that they still learn specific lexical patterns rather than the general principles of comparison.
Code (0)
등록된 구현이 없습니다.
Tasks
coreference-resolutionCoreference ResolutioncounterfactualNatural Language UnderstandingReading ComprehensionSimilar Papers 제목 키워드 기반
Machine Reading, Fast and Slow: When Do Models "Understand" Language?
Two of the most fundamental challenges in Natural Language Understanding (NLU) at present are: (a) how to establish whether deep learning-based models score highly on NLU benchmarks for the 'right' reasons; and (b) to un…
coreference-resolutionCoreference ResolutioncounterfactualNatural Language Understanding+1Asymptotic spreading of predator-prey populations in a shifting environment
Inspired by recent studies associating shifting temperature conditions with changes in the efficiency of predator species in converting their prey to offspring, we propose a predator-prey model of reaction-diffusion type…
Making Neural Machine Reading Comprehension Faster
This study aims at solving the Machine Reading Comprehension problem where questions have to be answered given a context passage. The challenge is to develop a computationally faster model which will have improved infere…
Knowledge DistillationMachine Reading ComprehensionNatural Language UnderstandingReading ComprehensionQANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
Current end-to-end machine reading and question answering (Q\&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inf…
Machine TranslationQuestion AnsweringReading ComprehensionTranslationGlobal Convergence and Geometric Characterization of Slow to Fast Weight Evolution in Neural Network Training for Classifying Linearly Non-Separable Data
In this paper, we study the dynamics of gradient descent in learning neural networks for classification problems. Unlike in existing works, we consider the linearly non-separable case where the training data of different…
General Classification