Understanding the Logical Capabilities of Large Language Models via Out-of-Context Representation Learning
We study the capabilities of Large Language Models (LLM) on binary relations, a ubiquitous concept in math employed in most reasoning, math and logic benchmarks. This work focuses on equality, inequality, and inclusion, along with the properties they satisfy, such as ir/reflexivity, a/symmetry, transitivity, and logical complexity (e.g., number of reasoning ``hops''). We propose an alternative to in-context learning that trains only the representations of newly introduced tokens, namely out-of-context representation learning. This method mitigates linguistic biases already present in a model and, differently from in-context learning, does not rely on external information or illustrations. We argue out-of-context representation learning as a better alternative to in-context learning and fine-tuning to evaluate the capabilities of LLMs on logic tasks that are the building blocks of more complex reasoning benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
In-Context LearningMathRepresentation LearningSimilar Papers 제목 키워드 기반
Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series
Recently, leveraging pre-trained Large Language Models (LLMs) for time series (TS) tasks has gained increasing attention, which involves activating and enhancing LLMs' capabilities. Many methods aim to activate LLMs' cap…
Time SeriesDisentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities
This study intends to systematically disentangle pure logic reasoning and text understanding by investigating the contrast across abstract and contextualized logical problems from a comprehensive set of domains. We explo…
Language ModelingLanguage ModellingLarge Language ModelLogical ReasoningHumanOmniV2: From Understanding to Omni-Modal Reasoning with Context
With the rapid evolution of multimodal large language models, the capacity to deeply understand and interpret human intentions has emerged as a critical capability, which demands detailed and thoughtful reasoning. In rec…
Large Language ModelMultimodal ReasoningReinforcement Learning (RL)CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment
Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. We introduce CoDA21 (Context Definition Alignment), a challenging benchmark that measures natur…
Natural Language UnderstandingWorld KnowledgeCoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment
Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. We introduce CoDA21 (Context Definition Alignment), a challenging benchmark that measures natur…
Natural Language UnderstandingWorld Knowledge