Logical forms complement probability in understanding language model (and human) performance
With the increasing interest in using large language models (LLMs) for planning in natural language, understanding their behaviors becomes an important research question. This work conducts a systematic investigation of LLMs' ability to perform logical reasoning in natural language. We introduce a controlled dataset of hypothetical and disjunctive syllogisms in propositional and modal logic and use it as the testbed for understanding LLM performance. Our results lead to novel insights in predicting LLM behaviors: in addition to the probability of input (Gonen et al., 2023; McCoy et al., 2024), logical forms should be considered as important factors. In addition, we show similarities and discrepancies between the logical reasoning performances of humans and LLMs by collecting and comparing behavioral data from both.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLogical ReasoningNatural Language UnderstandingSimilar Papers 제목 키워드 기반
Formulation of probability theory problem with subtle condition
Problems in probability theory prove to be one of the most challenging for students. Here, we formulate and discuss four related problems in probability theory that proved difficult for first to fourth-year undergraduate…
Complementary Strategies for Low Resourced Morphological Modeling
Morphologically rich languages are challenging for natural language processing tasks due to data sparsity. This can be addressed either by introducing out-of-context morphological knowledge, or by developing machine lear…
Morphological AnalysisWord EmbeddingsKnowledge Vector of Logical Reasoning in Large Language Models
Logical reasoning serve as a central capability in LLMs and includes three main forms: deductive, inductive, and abductive reasoning. In this work, we study the knowledge representations of these reasoning types in LLMs …
Logical ReasoningA Neural-Symbolic Approach to Natural Language Understanding
Deep neural networks, empowered by pre-trained language models, have achieved remarkable results in natural language understanding (NLU) tasks. However, their performances can drastically deteriorate when logical reasoni…
Logical ReasoningNatural Language InferenceNatural Language UnderstandingQuestion AnsweringEmbers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that in order to develop a holistic understanding of these systems we need to consider the…