How effective is BERT without word ordering? Implications for language understanding and data privacy
Ordered word sequences contain the rich structures that define language. However, it{'}s often not clear if or how modern pretrained language models utilize these structures. We show that the token representations and self-attention activations within BERT are surprisingly resilient to shuffling the order of input tokens, and that for several GLUE language understanding tasks, shuffling only minimally degrades performance, e.g., by 4{\%} for QNLI. While bleak from the perspective of language understanding, our results have positive implications for cases where copyright or ethics necessitates the consideration of bag-of-words data (vs. full documents). We simulate such a scenario for three sensitive classification tasks, demonstrating minimal performance degradation vs. releasing full language sequences.
Code (0)
등록된 구현이 없습니다.
Tasks
EthicsQNLIMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Word Reordering for Zero-shot Cross-lingual Structured Prediction
Adapting word order from one language to another is a key problem in cross-lingual structured prediction. Current sentence encoders (e.g., RNN, Transformer with position embeddings) are usually word order sensitive. Even…
Dependency ParsingMorphological TaggingPOSPosition+4Word Ordering Without Syntax
Recent work on word ordering has argued that syntactic structure is important, or even required, for effectively recovering the order of a sentence. We find that, in fact, an n-gram language model with a simple heuristic…
Language ModelingLanguage ModellingSentenceBERT-enhanced Relational Sentence Ordering Network
In this paper, we introduce a novel BERT-enhanced Relational Sentence Ordering Network (referred to as BRSON) by leveraging BERT for capturing better dependency relationship among sentences to enhance the coherence model…
DecoderSentenceSentence OrderingWord Alignment-Based Reordering of Source Chunks in PB-SMT
Reordering poses a big challenge in statistical machine translation between distant language pairs. The paper presents how reordering between distant language pairs can be handled efficiently in phrase-based statistical …
Machine TranslationTranslationWord AlignmentHilbert curves for efficient exploratory landscape analysis neighbourhood sampling
Landscape analysis aims to characterise optimisation problems based on their objective (or fitness) function landscape properties. The problem search space is typically sampled, and various landscape features are estimat…