Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?
Do state-of-the-art natural language understanding models care about word order - one of the most important characteristics of a sequence? Not always! We found 75% to 90% of the correct predictions of BERT-based classifiers, trained on many GLUE tasks, remain constant after input words are randomly shuffled. Despite BERT embeddings are famously contextual, the contribution of each individual word to downstream tasks is almost unchanged even after the word's context is shuffled. BERT-based models are able to exploit superficial cues (e.g. the sentiment of keywords in sentiment analysis; or the word-wise similarity between sequence-pair inputs in natural language inference) to make correct decisions when tokens are arranged in random orders. Encouraging classifiers to capture word order information improves the performance on most GLUE tasks, SQuAD 2.0 and out-of-samples. Our work suggests that many GLUE tasks are not challenging machines to understand the meaning of a sentence.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceNatural Language UnderstandingSentenceSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning to Organize a Bag of Words into Sentences with Neural Networks: An Empirical Study
Sequential information, a.k.a., orders, is assumed to be essential for processing a sequence with recurrent neural network or convolutional neural network based encoders. However, is it possible to encode natural languag…
SentenceDetecting Media Bias in News Articles using Gaussian Bias Distributions
Media plays an important role in shaping public opinion. Biased media can influence people in undesirable directions and hence should be unmasked as such. We observe that featurebased and neural text classification appro…
ArticlesBias DetectionSentencetext-classification+1Depth-Adaptive Graph Recurrent Network for Text Classification
The Sentence-State LSTM (S-LSTM) is a powerful and high efficient graph recurrent network, which views words as nodes and performs layer-wise recurrent steps between them simultaneously. Despite its successes on text rep…
ClassificationGeneral ClassificationSentencetext-classification+1Generalized Optimal Linear Orders
The sequential structure of language, and the order of words in a sentence specifically, plays a central role in human language processing. Consequently, in designing computational models of language, the de facto approa…
SentenceSentence Level Recurrent Topic Model: Letting Topics Speak for Themselves
We propose Sentence Level Recurrent Topic Model (SLRTM), a new topic model that assumes the generation of each word within a sentence to depend on both the topic of the sentence and the whole history of its preceding wor…
SentenceShort-Text ConversationTopic Models