Do Transformer Networks Improve the Discovery of Rules from Text?
With their Discovery of Inference Rules from Text (DIRT) algorithm, Lin and Pantel (2001) made a seminal contribution to the field of rule acquisition from text, by adapting the distributional hypothesis of Harris (1954) to rules that model binary relations such as X treat Y. DIRT’s relevance is renewed in today’s neural era given the recent focus on interpretability in the field of natural language processing. We propose a novel take on the DIRT algorithm, where we implement the distributional hypothesis using the contextualized embeddings provided by BERT, a transformer-network-based language model (Vaswani et al. 2017; Devlin et al. 2018). In particular, we change the similarity measure between pairs of slots (i.e., the set of words matched by a rule) from the original formula that relies on lexical items to a formula computed using contextualized embeddings. We empirically demonstrate that this new similarity method yields a better implementation of the distributional hypothesis, and this, in turn, yields rules that outperform the original algorithm in the question answering-based evaluation proposed by Lin and Pantel (2001).
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingQuestion AnsweringSimilar Papers 제목 키워드 기반
Neural-Guided Equation Discovery
Deep learning approaches are becoming increasingly attractive for equation discovery. We show the advantages and disadvantages of using neural-guided equation discovery by giving an overview of recent papers and the resu…
Contrastive LearningEquation Discoveryreinforcement-learningReinforcement LearningAbductionRules: Training Transformers to Explain Unexpected Inputs
Transformers have recently been shown to be capable of reliably performing logical reasoning over facts and rules expressed in natural language, but abductive reasoning - inference to the best explanation of an unexpecte…
Common Sense ReasoningLogical Reasoningscientific discoveryAbductionRules: Training Transformers to Explain Unexpected Inputs
Transformers have recently been shown to be capable of reliably performing logical reasoning over facts and rules expressed in natural language, but abductive reasoning - inference to the best explanation of an unexpecte…
Common Sense ReasoningLogical Reasoningscientific discoveryAutomatic Rule Induction for Interpretable Semi-Supervised Learning
Semi-supervised learning has shown promise in allowing NLP models to generalize from small amounts of labeled data. Meanwhile, pretrained transformer models act as black-box correlation engines that are difficult to expl…
Relation ExtractionOnline false discovery rate control for anomaly detection in time series
This article proposes novel rules for false discovery rate control (FDRC) geared towards online anomaly detection in time series. Online FDRC rules allow to control the properties of a sequence of statistical tests. In t…
Anomaly DetectionTime SeriesTime Series Analysis