AbductionRules: Training Transformers to Explain Unexpected Inputs
Transformers have recently been shown to be capable of reliably performing logical reasoning over facts and rules expressed in natural language, but abductive reasoning - inference to the best explanation of an unexpected observation - has been underexplored despite significant applications to scientific discovery, common-sense reasoning, and model interpretability. This paper presents AbductionRules, a group of natural language datasets designed to train and test generalisable abduction over natural-language knowledge bases. We use these datasets to finetune pretrained Transformers and discuss their performance, finding that our models learned generalisable abductive techniques but also learned to exploit the structure of our data. Finally, we discuss the viability of this approach to abductive reasoning and ways in which it may be improved in future work.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningLogical Reasoningscientific discoverySimilar Papers 제목 키워드 기반
AbductionRules: Training Transformers to Explain Unexpected Inputs
Transformers have recently been shown to be capable of reliably performing logical reasoning over facts and rules expressed in natural language, but abductive reasoning - inference to the best explanation of an unexpecte…
Common Sense ReasoningLogical Reasoningscientific discoveryShattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic
Large language models (LLMs) often exhibit unexpected errors or unintended behavior, even at scale. While recent work reveals the discrepancy between LLMs and humans in skill compositions, the learning dynamics of skill …
Learning the greatest common divisor: explaining transformer predictions
The predictions of small transformers, trained to calculate the greatest common divisor (GCD) of two positive integers, can be fully characterized by looking at model inputs and outputs. As training proceeds, the model l…
Mitigating the Likelihood Paradox in Flow-based OOD Detection via Entropy Manipulation
Deep generative models that can tractably compute input likelihoods, including normalizing flows, often assign unexpectedly high likelihoods to out-of-distribution (OOD) inputs. We mitigate this likelihood paradox by man…
Semantic SimilarityJoint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4
Modern high-scoring models of vision in the brain score competition do not stem from Vision Transformers. However, in this paper, we provide evidence against the unexpected trend of Vision Transformers (ViT) being not pe…
Adversarial Robustness