BERT & Family Eat Word Salad: Experiments with Text Understanding

2021-01-10 · Ashim Gupta, Giorgi Kvernadze, Vivek Srikumar

In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.

📄 PDF Abstract BibTeX arXiv:2101.03453

Code (1)

utahnlp/word-salad 공식 구현

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Weight Decay 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Multi-Head Attention 설명 없음

Attention 설명 없음

WordPiece 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Word Salad Chopper: Reasoning Models Waste A Ton Of Decoding Budget On Useless Repetitions, Self-Knowingly

2025-11-01 · Wenya Xie, Shaochen, Zhong, Hoang Anh Duy Le 외 arxiv

Large Reasoning Models (LRMs) are often bottlenecked by the high cost of output tokens. We show that a significant portion of these tokens are useless self-repetitions - what we call "word salad" - that exhaust the decod…

Picking Apart Story Salads

2018-10-31 · EMNLP 2018 10 · Su Wang, Eric Holgate, Greg Durrett, Katrin Erk

During natural disasters and conflicts, information about what happened is often confusing, messy, and distributed across many sources. We would like to be able to automatically identify relevant information and assemble…

Clustering

SALAD: Skeleton-aware Latent Diffusion for Text-driven Motion Generation and Editing

2025-03-18 · CVPR 2025 1 · Seokhyeon Hong, Chaelin Kim, Serin Yoon, Junghyun Nam 외

Text-driven motion generation has advanced significantly with the rise of denoising diffusion models. However, previous methods often oversimplify representations for the skeletal joints, temporal frames, and textual wor…

Denoising Motion Generation

Word Salad: Relating Food Prices and Descriptions

2012-07-01 · EMNLP 2012 7 · Victor Chahuneau, Kevin Gimpel, Bryan R. Routledge, Lily Scherlis 외

CEFER: A Four Facets Framework based on Context and Emotion embedded features for Implicit and Explicit Emotion Recognition

2022-09-28 · Fereshteh Khoshnam, Ahmad Baraani-Dastjerdi, M. J. Liaghatdar

People's conduct and reactions are driven by their emotions. Online social media is becoming a great instrument for expressing emotions in written form. Paying attention to the context and the entire sentence help us to …

Emotion Recognition Sentence