Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation
Following the recent success of word embeddings, it has been argued that there is no such thing as an ideal representation for words, as different models tend to capture divergent and often mutually incompatible aspects like semantics/syntax and similarity/relatedness. In this paper, we show that each embedding model captures more information than directly apparent. A linear transformation that adjusts the similarity order of the model without any external resource can tailor it to achieve better results in those aspects, providing a new perspective on how embeddings encode divergent linguistic information. In addition, we explore the relation between intrinsic and extrinsic evaluation, as the effect of our transformations in downstream tasks is higher for unsupervised systems than for supervised ones.
Code (2)
Tasks
Word EmbeddingsSimilar Papers 제목 키워드 기반
Field Embedding: A Unified Grain-Based Framework for Word Representation
Word representations empowered with additional linguistic information have been widely studied and proved to outperform traditional embeddings. Current methods mainly focus on learning embeddings for words while embeddin…
Word EmbeddingsWhat company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris
The power of word embeddings is attributed to the linguistic theory that similar words will appear in similar contexts. This idea is specifically invoked by noting that "you shall know a word by the company it keeps," a …
Word EmbeddingsWhat you can cram into a single vector: Probing sentence embeddings for linguistic properties
Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are …
General ClassificationSentenceSentence ClassificationSentence EmbeddingsWhat you can cram into a single \$\&!\#* vector: Probing sentence embeddings for linguistic properties
Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. {``}Downstream{''} tasks, often based on sentence classification…
General ClassificationMachine TranslationSentenceSentence Classification+2Uncovering Probabilistic Implications in Typological Knowledge Bases
The study of linguistic typology is rooted in the implications we find between linguistic features, such as the fact that languages with object-verb word ordering tend to have post-positions. Uncovering such implications…
Knowledge Base Population