How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets
A central question in natural language understanding (NLU) research is whether high performance demonstrates the models' strong reasoning capabilities. We present an extensive series of controlled experiments where pre-trained language models are exposed to data that have undergone specific corruption transformations. These involve removing instances of specific word classes and often lead to non-sensical sentences. Our results show that performance remains high on most GLUE tasks when the models are fine-tuned or tested on corrupted data, suggesting that they leverage other cues for prediction even in non-sensical contexts. Our proposed data transformations can be used to assess the extent to which a specific dataset constitutes a proper testbed for evaluating models' language understanding capabilities.
Code (1)
Tasks
Natural Language UnderstandingSimilar Papers 제목 키워드 기반
How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets
A central question in natural language understanding (NLU) research is whether high performance demonstrates the models' strong reasoning capabilities. We present an extensive series of controlled experiments where pre-t…
Natural Language UnderstandingLost in Transmission: On the Impact of Networking Corruptions on Video Machine Learning Models
We study how networking corruptions--data corruptions caused by networking errors--affect video machine learning (ML) models. We discover apparent networking corruptions in Kinetics-400, a benchmark video ML dataset. In …
Data AugmentationFrom the Top Down: Does Corruption Affect Performance?
Corruption, fraud, and unethical activities have emerged as significant obstacles to global economic, political, and social progress. Although many empirical studies have focused on country-level corruption metrics, this…
CorrGAN: Input Transformation Technique Against Natural Corruptions
Because of the increasing accuracy of Deep Neural Networks (DNNs) on different tasks, a lot of real times systems are utilizing DNNs. These DNNs are vulnerable to adversarial perturbations and corruptions. Specifically, …
Generative Adversarial NetworkModels in the Wild: On Corruption Robustness of NLP Systems
Natural Language Processing models lack a unified approach to robustness testing. In this paper we introduce WildNLP - a framework for testing model stability in a natural setting where text corruptions such as keyboard …
NERSentiment AnalysisWord Embeddings