What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models
Recent work has argued that large language models (LLMs) are not "abstract reasoners", citing their poor zero-shot performance on a variety of challenging tasks as evidence. We revisit these experiments in order to add nuance to the claim. First, we show that while LLMs indeed perform poorly in a zero-shot setting, even tuning a small subset of parameters for input encoding can enable near-perfect performance. However, we also show that this finetuning does not necessarily transfer across datasets. We take this collection of empirical results as an invitation to (re-)open the discussion of what it means to be an "abstract reasoner", and why it matters whether LLMs fit the bill.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
An Extension-based Approach for Computing and Verifying Preferences in Abstract Argumentation
We present an extension-based approach for computing and verifying preferences in an abstract argumentation system. Although numerous argumentation semantics have been developed previously for identifying acceptable sets…
Abstract ArgumentationRevisiting initial sets in abstract argumentation
We revisit the notion of initial sets by Xu and Cayrol, i.e., non-empty minimal admissible sets in abstract argumentation frameworks. Initial sets are a simple concept for analysing conflicts in an abstract argumentation…
Abstract ArgumentationBeyond But-for Test: Counterfactual Explanation in Abstract Argumentation via Actual Causality (Extended Version)
Counterfactual explanation in abstract argumentation calls for an answer to the what-if query: would the topic argument still be accepted if the status of certain other arguments were changed? Existing approaches are lim…
The Diversity of Argument-Making in the Wild: from Assumptions and Definitions to Causation and Anecdote in Reddit's "Change My View"
What kinds of arguments do people make, and what effect do they have on others? Normative constraints on argument-making are as old as philosophy itself, but little is known about the diversity of arguments made in pract…
DiversityPhilosophy