Limit Analysis for Symbolic Multi-step Reasoning Tasks with Information Propagation Rules Based on Transformers
Transformers are able to perform reasoning tasks, however the intrinsic mechanism remains widely open. In this paper we propose a set of information propagation rules based on Transformers and utilize symbolic reasoning tasks to theoretically analyze the limit reasoning steps. We show that the limit number of reasoning steps is between $O(3^{L-1})$ and $O(2^{L-1})$ for a model with $L$ attention layers in a single-pass.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Revealing Algorithmic Deductive Circuits for Logical Reasoning
Recent studies have shown that Large Language Models (LLMs) can achieve strong reasoning performance by incorporating functional symbolic representations that abstractly describe graph traversal algorithms and step-by-st…
Few-Shot LearningLogical ReasoningSpatialMath: Spatial Comprehension-Infused Symbolic Reasoning for Mathematical Problem-Solving
Multimodal Small-to-Medium sized Language Models (MSLMs) have demonstrated strong capabilities in integrating visual and textual information but still face significant limitations in visual comprehension and mathematical…
Mathematical ReasoningData AugmentationSymbolic Working Memory Enhances Language Models for Complex Rule Application
Large Language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning involving a series of rule application steps, especially when rules are presented non-sequentially…
Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation
Symbolic Music, akin to language, can be encoded in discrete symbols. Recent research has extended the application of large language models (LLMs) such as GPT-4 and Llama2 to the symbolic music domain including understan…
H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables
Tabular reasoning involves interpreting natural language queries about tabular data, which presents a unique challenge of combining language understanding with structured data analysis. Existing methods employ either tex…
Fact VerificationMathematical ReasoningNatural Language QueriesQuestion Answering+2