Does Entity Abstraction Help Generative Transformers Reason?

2022-01-05 · Nicolas Gontier, Siva Reddy, Christopher Pal

We study the utility of incorporating entity type abstractions into pre-trained Transformers and test these methods on four NLP tasks requiring different forms of logical reasoning: (1) compositional language understanding with text-based relational reasoning (CLUTRR), (2) abductive reasoning (ProofWriter), (3) multi-hop question answering (HotpotQA), and (4) conversational question answering (CoQA). We propose and empirically explore three ways to add such abstraction: (i) as additional input embeddings, (ii) as a separate sequence to encode, and (iii) as an auxiliary prediction task for the model. Overall, our analysis demonstrates that models with abstract entity knowledge performs better than without it. The best abstraction aware models achieved an overall accuracy of 88.8% and 91.8% compared to the baseline model achieving 62.9% and 89.8% on CLUTRR and ProofWriter respectively. However, for HotpotQA and CoQA, we find that F1 scores improve by only 0.5% on average. Our results suggest that the benefit of explicit abstraction is significant in formally defined logical reasoning settings requiring many reasoning hops, but point to the notion that it is less beneficial for NLP tasks having less formal logical structure.

📄 PDF Abstract BibTeX arXiv:2201.01787

Code (0)

등록된 구현이 없습니다.

Tasks

Conversational Question Answering Logical Reasoning Multi-hop Question Answering Question Answering Relational Reasoning

Methods 이 논문이 사용한 방법론

AWARE We propose to theoretically and empirically examine the effect of incorporating weighting schemes into walk-aggregating GNNs. To this end, we propose a simple, interpretable, and…

Transformer Explainer: Interactive Learning of Text-Generative Models

2024-08-08 · Aeree Cho, Grace C. Kim, Alexander Karpekov, Alec Helbling 외

Transformers have revolutionized machine learning, yet their inner workings remain opaque to many. We present Transformer Explainer, an interactive visualization tool designed for non-experts to learn about Transformers …

Entity Abstraction in Visual Model-Based Reinforcement Learning

2019-10-28 · Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner 외

This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a co…

model Model-based Reinforcement Learning Object Object Discovery+5

Swing-by Dynamics in Concept Learning and Compositional Generalization

2024-10-10 · Yongyi Yang, Core Francisco Park, Ekdeep Singh Lubana, Maya Okawa 외

Prior work has shown that text-conditioned diffusion models can learn to identify and manipulate primitive concepts underlying a compositional data-generating process, enabling generalization to entirely novel, out-of-di…

Odin: Oriented Dual-module Integration for Text-rich Network Representation Learning

2025-11-26 · Kaifeng Hong, Yinglong Zhang, Xiaoying Hong, Xuewen Xia 외 arxiv

Text-attributed graphs require models to effectively combine strong textual understanding with structurally informed reasoning. Existing approaches either rely on GNNs--limited by over-smoothing and hop-dependent diffusi…

Representation Learning

Abstraction and Relational learning

2009-12-01 · NeurIPS 2009 12 · Charles Kemp, Alan Jern

Many categories are better described by providing relational information than listing characteristic features. We present a hierarchical generative model that helps to explain how relational categories are learned and u…

Relational Reasoning