Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning
Recent years have seen impressive performance of transformer-based models on different natural language processing tasks. However, it is not clear to what degree the transformers can reason on natural language. To shed light on this question, this survey paper discusses the performance of transformers on different reasoning tasks, including mathematical reasoning, commonsense reasoning, and logical reasoning. We point out successes and limitations, of both empirical and theoretical nature.
Code (1)
Tasks
Deep LearningLogical ReasoningMathematical ReasoningSurveySimilar Papers 제목 키워드 기반
Chain-of-Thought Enhanced Shallow Transformers for Wireless Symbol Detection
Transformers have shown potential in solving wireless communication problems, particularly via in-context learning (ICL), where models adapt to new tasks through prompts without requiring model updates. However, prior IC…
Computational EfficiencyIn-Context LearningMulti-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent
Transformers have demonstrated remarkable capabilities in multi-step reasoning tasks. However, understandings of the underlying mechanisms by which they acquire these abilities through training remain limited, particular…
Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries
Knowledge graph (KG) embeddings have been a mainstream approach for reasoning over incomplete KGs. However, limited by their inherently shallow and static architectures, they can hardly deal with the rising focus on comp…
Mixture-of-ExpertsSelf-Verifying Reflection Helps Transformers with CoT Reasoning
Advanced large language models (LLMs) frequently reflect in reasoning chain-of-thoughts (CoTs), where they self-verify the correctness of current solutions and explore alternatives. However, given recent findings that LL…
Reinforcement LearningDynamic Reasoning Chains through Depth-Specialized Mixture-of-Experts in Transformer Architectures
Contemporary transformer architectures apply identical processing depth to all inputs, creating inefficiencies and limiting reasoning quality. Simple factual queries are subjected to the same multilayered computation as …