Logical Tasks for Measuring Extrapolation and Rule Comprehension
Logical reasoning is essential in a variety of human activities. A representative example of a logical task is mathematics. Recent large-scale models trained on large datasets have been successful in various fields, but their reasoning ability in arithmetic tasks is limited, which we reproduce experimentally. Here, we recast this limitation as not unique to mathematics but common to tasks that require logical operations. We then propose a new set of tasks, termed logical tasks, which will be the next challenge to address. This higher point of view helps the development of inductive biases that have broad impact beyond the solution of individual tasks. We define and characterize logical tasks and discuss system requirements for their solution. Furthermore, we discuss the relevance of logical tasks to concepts such as extrapolation, explainability, and inductive bias. Finally, we provide directions for solving logical tasks.
Code (1)
Tasks
Inductive BiasLogical ReasoningSimilar Papers 제목 키워드 기반
Deep Manifold Learning for Reading Comprehension and Logical Reasoning Tasks with Polytuplet Loss
The current trend in developing machine learning models for reading comprehension and logical reasoning tasks is focused on improving the models' abilities to understand and utilize logical rules. This work focuses on pr…
Logical ReasoningReading ComprehensionTransfer LearningFrom LSAT: The Progress and Challenges of Complex Reasoning
Complex reasoning aims to draw a correct inference based on complex rules. As a hallmark of human intelligence, it involves a degree of explicit reading comprehension, interpretation of logical knowledge and complex rule…
Few-Shot LearningLogical ReasoningReading ComprehensionCellARC: Measuring Intelligence with Cellular Automata
We introduce CellARC, a synthetic benchmark for abstraction and reasoning built from multicolor 1D cellular automata (CA). Each episode has five support pairs and one query serialized in 256 tokens, enabling rapid iterat…
Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts
LLMs show remarkable emergent abilities, such as inferring concepts from presumably out-of-distribution prompts, known as in-context learning. Though this success is often attributed to the Transformer architecture, our …
In-Context LearningState Space ModelsMastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks
Language models' (LMs) proficiency in handling deterministic symbolic reasoning and rule-based tasks remains limited due to their dependency implicit learning on textual data. To endow LMs with genuine rule comprehension…
Arithmetic ReasoningLanguage Modelling