Squashed Shifted PMI Matrix: Bridging Word Embeddings and Hyperbolic Spaces
We show that removing sigmoid transformation in the skip-gram with negative sampling (SGNS) objective does not harm the quality of word vectors significantly and at the same time is related to factorizing a squashed shifted PMI matrix which, in turn, can be treated as a connection probabilities matrix of a random graph. Empirically, such graph is a complex network, i.e. it has strong clustering and scale-free degree distribution, and is tightly connected with hyperbolic spaces. In short, we show the connection between static word embeddings and hyperbolic spaces through the squashed shifted PMI matrix using analytical and empirical methods.
Code (2)
Tasks
ClusteringWord EmbeddingsSimilar Papers 제목 키워드 기반
Neural Word Embedding as Implicit Matrix Factorization
We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information…
Word SimilarityEnhanced Word Representations for Bridging Anaphora Resolution
Most current models of word representations(e.g.,GloVe) have successfully captured fine-grained semantics. However, semantic similarity exhibited in these word embeddings is not suitable for resolving bridging anaphora, …
Bridging Anaphora ResolutionSemantic SimilaritySemantic Textual SimilarityWord EmbeddingsA Deterministic Algorithm for Bridging Anaphora Resolution
Previous work on bridging anaphora resolution (Poesio et al., 2004; Hou et al., 2013b) use syntactic preposition patterns to calculate word relatedness. However, such patterns only consider NPs' head nouns and hence do n…
Bridging Anaphora ResolutionWord EmbeddingsAttention Focusing for Neural Machine Translation by Bridging Source and Target Embeddings
In neural machine translation, a source sequence of words is encoded into a vector from which a target sequence is generated in the decoding phase. Differently from statistical machine translation, the associations betwe…
Machine TranslationSentenceTranslationWord EmbeddingsAcoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognition
Direct acoustics-to-word (A2W) systems for end-to-end automatic speech recognition are simpler to train, and more efficient to decode with, than sub-word systems. However, A2W systems can have difficulties at training ti…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech Recognition+1