Are distributional representations ready for the real world? Evaluating word vectors for grounded perceptual meaning
Distributional word representation methods exploit word co-occurrences to build compact vector encodings of words. While these representations enjoy widespread use in modern natural language processing, it is unclear whether they accurately encode all necessary facets of conceptual meaning. In this paper, we evaluate how well these representations can predict perceptual and conceptual features of concrete concepts, drawing on two semantic norm datasets sourced from human participants. We find that several standard word representations fail to encode many salient perceptual features of concepts, and show that these deficits correlate with word-word similarity prediction errors. Our analyses provide motivation for grounded and embodied language learning approaches, which may help to remedy these deficits.
Code (1)
Tasks
Word SimilaritySimilar Papers 제목 키워드 기반
Evaluating Predictive Uncertainty under Distributional Shift on Dialogue Dataset
In open-domain dialogues, predictive uncertainties are mainly evaluated in a domain shift setting to cope with out-of-distribution inputs. However, in real-world conversations, there could be more extensive distributiona…
Distributed Vector Representations for Unsupervised Automatic Short Answer Grading
We address the problem of automatic short answer grading, evaluating a collection of approaches inspired by recent advances in distributional text representations. In addition, we propose an unsupervised approach for det…
automatic short answer gradingReading Comprehensiontext similarityWord EmbeddingsDistributional Semantics in the Real World: Building Word Vector Representations from a Truth-Theoretic Model
Distributional semantics models (DSMs) are known to produce excellent representations of word meaning, which correlate with a range of behavioural data. As lexical representations, they have been said to be fundamentally…
Evaluating Predictive Uncertainty and Robustness to Distributional Shift Using Real World Data
Most machine learning models operate under the assumption that the training, testing and deployment data is independent and identically distributed (i.i.d.). This assumption doesn't generally hold true in a natural setti…
Direct vs. indirect evaluation of distributional thesauri
With the success of word embedding methods in various Natural Language Processing tasks, all the field of distributional semantics has experienced a renewed interest. Beside the famous word2vec, recent studies have prese…
Information RetrievalRetrieval