Using Distributional Principles for the Semantic Study of Contextual Language Models
Many studies were recently done for investigating the properties of contextual language models but surprisingly, only a few of them consider the properties of these models in terms of semantic similarity. In this article, we first focus on these properties for English by exploiting the distributional principle of substitution as a probing mechanism in the controlled context of SemCor and WordNet paradigmatic relations. Then, we propose to adapt the same method to a more open setting for characterizing the differences between static and contextual language models.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic SimilaritySemantic Textual SimilaritySimilar Papers 제목 키워드 기반
Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture
This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into …
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model
Contextualized word embeddings, i.e. vector representations for words in context, are naturally seen as an extension of previous noncontextual distributional semantic models. In this work, we focus on BERT, a deep neural…
PositionSentenceWord EmbeddingsBuilding Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri?
While contextual language models are now dominant in the field of Natural Language Processing, the representations they build at the token level are not always suitable for all uses. In this article, we propose a new met…
Semantic SimilaritySemantic Textual SimilarityVocal Bursts Type PredictionA comparative evaluation and analysis of three generations of Distributional Semantic Models
Distributional semantics has deeply changed in the last decades. First, predict models stole the thunder from traditional count ones, and more recently both of them were replaced in many NLP applications by contextualize…
A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces
We study semantic construal in grammatical constructions using large language models. First, we project contextual word embeddings into three interpretable semantic spaces, each defined by a different set of psycholingui…
ObjectPositionSentenceWord Embeddings