Evaluating Compositionality in Sentence Embeddings
An important challenge for human-like AI is compositional semantics. Recent research has attempted to address this by using deep neural networks to learn vector space embeddings of sentences, which then serve as input to other tasks. We present a new dataset for one such task, `natural language inference' (NLI), that cannot be solved using only word-level knowledge and requires some compositionality. We find that the performance of state of the art sentence embeddings (InferSent; Conneau et al., 2017) on our new dataset is poor. We analyze the decision rules learned by InferSent and find that they are consistent with simple heuristics that are ecologically valid in its training dataset. Further, we find that augmenting training with our dataset improves test performance on our dataset without loss of performance on the original training dataset. This highlights the importance of structured datasets in better understanding and improving AI systems.
Code (1)
Tasks
Natural Language InferenceSentenceSentence EmbeddingsvalidSimilar Papers 제목 키워드 기반
Evaluating Compositionality of Sentence Representation Models
We evaluate the compositionality of general-purpose sentence encoders by proposing two different metrics to quantify compositional understanding capability of sentence encoders. We introduce a novel metric, Polarity Sens…
SensitivitySentenceQuantifying Compositionality of Classic and State-of-the-Art Embeddings
For language models to generalize correctly to novel expressions, it is critical that they exploit access compositional meanings when this is justified. Even if we don't know what a "pelp" is, we can use our knowledge of…
Knowledge GraphsComparing word2vec and GloVe for Automatic Measurement of MWE Compositionality
This paper explores the use of word2vec and GloVe embeddings for unsupervised measurement of the semantic compositionality of MWE candidates. Through comparison with several human-annotated reference sets, we find word2v…
Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings
Past work probing compositionality in sentence embedding models faces issues determining the causal impact of implicit syntax representations. Given a sentence, we construct a neural module net based on its syntax parse …
Semantic CompositionSentenceSentence EmbeddingSentence-Embedding+1Measuring Compositionality in Representation Learning
Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to…
BIG-bench Machine LearningRepresentation Learning