Does BERT Understand Idioms? A Probing-Based Empirical Study of BERT Encodings of Idioms
Understanding idioms is important in NLP. In this paper, we study to what extent pre-trained BERT model can encode the meaning of a potentially idiomatic expression (PIE) in a certain context. We make use of a few existing datasets and perform two probing tasks: PIE usage classification and idiom paraphrase identification. Our experiment results suggest that BERT indeed can separate the literal and idiomatic usages of a PIE with high accuracy. It is also able to encode the idiomatic meaning of a PIE to some extent.
Code (0)
등록된 구현이 없습니다.
Tasks
Paraphrase IdentificationSimilar Papers 제목 키워드 기반
Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space
The goal of this paper is to learn more about how idiomatic information is structurally encoded in embeddings, using a structural probing method. We repurpose an existing English verbal multi-word expression (MWE) datase…
Open-Ended Question AnsweringFinding BERT’s Idiomatic Key
Sentence embeddings encode information relating to the usage of idioms in a sentence. This paper reports a set of experiments that combine a probing methodology with input masking to analyse where in a sentence this idio…
SentenceSentence EmbeddingsUnderstanding Transformer Memorization Recall Through Idioms
To produce accurate predictions, language models (LMs) must balance between generalization and memorization. Yet, little is known about the mechanism by which transformer LMs employ their memorization capacity. When does…
MemorizationOn the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT
Contextualized word representations have become a driving force in NLP, motivating widespread interest in understanding their capabilities and the mechanisms by which they operate. Particularly intriguing is their abilit…
Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntax
This study analyzes the attention patterns of fine-tuned encoder-only models based on the BERT architecture (BERT-based models) towards two distinct types of Multiword Expressions (MWEs): idioms and microsyntactic units …