Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning
Pretrained masked language models (PLMs) were shown to be inheriting a considerable amount of relational knowledge from the source corpora. In this paper, we present an in-depth and comprehensive study concerning specializing PLMs into relational models from the perspective of network pruning. We show that it is possible to find subnetworks capable of representing grounded commonsense relations at non-trivial sparsity while being more generalizable than original PLMs in scenarios requiring knowledge of single or multiple commonsense relations.
Code (1)
Tasks
Network PruningRelational ReasoningSimilar Papers 제목 키워드 기반
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering
Existing work on augmenting question answering (QA) models with external knowledge (e.g., knowledge graphs) either struggle to model multi-hop relations efficiently, or lack transparency into the model's prediction ratio…
Knowledge GraphsQuestion AnsweringRelational ReasoningRelation NetworkLearning to Reason With Relational Abstractions
Large language models have recently shown promising progress in mathematical reasoning when fine-tuned with human-generated sequences walking through a sequence of solution steps. However, the solution sequences are not …
Mathematical ReasoningNumerical Reasoning over Legal Contracts via Relational Database
Numerical reasoning over text requires deep integration between the semantic understanding of the natural language context and the mathematical calculation of the symbolic terms. However, existing approaches are limited …
Agentic AI and the next intelligence explosion
The "AI singularity" is often miscast as a monolithic, godlike mind. Evolution suggests a different path: intelligence is fundamentally plural, social, and relational. Recent advances in agentic AI reveal that frontier r…
Probing Cross-Modal Representations in Multi-Step Relational Reasoning
We investigate the representations learned by vision and language models in tasks that require relational reasoning. Focusing on the problem of assessing the relative size of objects in abstract visual contexts, we analy…
DiagnosticRelational Reasoning