LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning

2021-01-15 · Yuhuai Wu, Markus Rabe, Wenda Li, Jimmy Ba, Roger Grosse, Christian Szegedy

While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are flexible enough to learn inductive bias from suitable generic tasks. Here, we replace architecture engineering by encoding inductive bias in the form of datasets. Inspired by Peirce's view that deduction, induction, and abduction are the primitives of reasoning, we design three synthetic tasks that are intended to require the model to have these three abilities. We specifically design these tasks to be synthetic and devoid of mathematical knowledge to ensure that only the fundamental reasoning biases can be learned from these tasks. This defines a new pre-training methodology called "LIME" (Learning Inductive bias for Mathematical rEasoning). Models trained with LIME significantly outperform vanilla transformers on four very different large mathematical reasoning benchmarks. Unlike dominating the computation cost as traditional pre-training approaches, LIME requires only a small fraction of the computation cost of the typical downstream task. The code for generating LIME tasks is available at https://github.com/tonywu95/LIME.

📄 PDF Abstract BibTeX arXiv:2101.06223

Code (1)

tonywu95/lime 공식 구현

Tasks

Inductive BiasMathematical Reasoning

Methods 이 논문이 사용한 방법론

LIME LIME, or Local Interpretable Model-Agnostic Explanations, is an algorithm that can explain the predictions of any classifier or regressor in a faithful way, by…

On the Inductive Bias of Stacking Towards Improving Reasoning

2024-09-27 · Nikunj Saunshi, Stefani Karp, Shankar Krishnan, Sobhan Miryoosefi 외

Given the increasing scale of model sizes, novel training strategies like gradual stacking [Gong et al., 2019, Reddi et al., 2023] have garnered interest. Stacking enables efficient training by gradually growing the dept…

Inductive Bias Math Reading Comprehension

Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines

2022-05-23 · Sreejan Kumar, Carlos G. Correa, Ishita Dasgupta, Raja Marjieh 외

Strong inductive biases give humans the ability to quickly learn to perform a variety of tasks. Although meta-learning is a method to endow neural networks with useful inductive biases, agents trained by meta-learning ma…

Meta-Learning Meta Reinforcement Learning Program induction Reinforcement Learning (RL)

Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in Transformers

2023-04-01 · Awni Altabaa, Taylor Webb, Jonathan Cohen, John Lafferty

An extension of Transformers is proposed that enables explicit relational reasoning through a novel module called the Abstractor. At the core of the Abstractor is a variant of attention called relational cross-attention.…

Inductive Bias Mathematical Problem-Solving Relational Reasoning

Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?

2026-04-23 · Shivam Rawat, Lucie Flek, Florian Mai, Nicholas Kluge Corrêa arxiv

Reasoning in large language models is often discussed as a single capability, but some of its gains may stem from simpler underlying operations. We examine two such primitives, recall and state-tracking, through five con…

InductionBench: LLMs Fail in the Simplest Complexity Class

2025-02-20 · Wenyue Hua, Tyler Wong, Sun Fei, Liangming Pan 외

Large language models (LLMs) have shown remarkable improvements in reasoning and many existing benchmarks have been addressed by models such as o1 and o3 either fully or partially. However, a majority of these benchmarks…

scientific discovery