Why does PairDiff work? - A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection
Representing the semantic relations that exist between two given words (or entities) is an important first step in a wide-range of NLP applications such as analogical reasoning, knowledge base completion and relational information retrieval. A simple, yet surprisingly accurate method for representing a relation between two words is to compute the vector offset (PairDiff) between their corresponding word embeddings. Despite the empirical success, it remains unclear as to whether PairDiff is the best operator for obtaining a relational representation from word embeddings. We conduct a theoretical analysis of generalised bilinear operators that can be used to measure the l2 relational distance between two word-pairs. We show that, if the word embed- dings are standardised and uncorrelated, such an operator will be independent of bilinear terms, and can be simplified to a linear form, where PairDiff is a special case. For numerous word embedding types, we empirically verify the uncorrelation assumption, demonstrating the general applicability of our theoretical result. Moreover, we experimentally discover PairDiff from the bilinear relational compositional operator on several benchmark analogy datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalKnowledge Base CompletionRetrievalWord EmbeddingsSimilar Papers 제목 키워드 기반
Why PairDiff works? -- A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection
Representing the semantic relations that exist between two given words (or entities) is an important first step in a wide-range of NLP applications such as analogical reasoning, knowledge base completion and relational i…
Information RetrievalKnowledge Base CompletionRelationRetrieval+1A technical note on bilinear layers for interpretability
The ability of neural networks to represent more features than neurons makes interpreting them challenging. This phenomenon, known as superposition, has spurred efforts to find architectures that are more interpretable t…
BERT is not The Count: Learning to Match Mathematical Statements with Proofs
We introduce a task consisting in matching a proof to a given mathematical statement. The task fits well within current research on Mathematical Information Retrieval and, more generally, mathematical article analysis (M…
ArticlesInformation RetrievalRetrievalCompact Bilinear Pooling
Bilinear models has been shown to achieve impressive performance on a wide range of visual tasks, such as semantic segmentation, fine grained recognition and face recognition. However, bilinear features are high dimensio…
Face RecognitionFew-Shot LearningGeneral Classificationimage-classification+2Koopman System Approximation Based Optimal Control of Multiple Robots -- Part I: Concepts and Formulations
This paper presents a study of the Koopman operator theory and its application to optimal control of a multi-robot system. The Koopman operator, while operating on a set of observation functions of the state vector of a …