POE: Process of Elimination for Multiple Choice Reasoning
Language models (LMs) are capable of conducting in-context learning for multiple choice reasoning tasks, but the options in these tasks are treated equally. As humans often first eliminate wrong options before picking the final correct answer, we argue a similar two-step strategy can make LMs better at these tasks. To this end, we present the Process of Elimination (POE), a two-step scoring method. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options. Zero-shot experiments on 8 reasoning tasks illustrate the effectiveness of POE, and a following analysis finds our method to be especially performant on logical reasoning tasks. We further analyze the effect of masks, and show that POE applies to few-shot settings and large language models (LLMs) like ChatGPT.
Code (1)
Tasks
In-Context LearningLogical ReasoningMultiple-choiceSimilar Papers 제목 키워드 기반
MM-PoE: Multiple Choice Reasoning via. Process of Elimination using Multi-Modal Models
This paper introduces Multiple Choice Reasoning via. Process of Elimination using Multi-Modal models, herein referred to as Multi-Modal Process of Elimination (MM-PoE). This novel methodology is engineered to augment the…
Multiple-choiceQuestion AnsweringVisual Question AnsweringVisual ReasoningIt's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning
Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used wit…
Multiple-choiceRethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design
Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiab…
Reinforcement LearningEvaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
To precisely evaluate a language model's capability for logical reading comprehension, we present a dataset for testing the understanding of the rationale behind critical reasoning. For questions taken from an existing m…
Multiple-choiceReading ComprehensionOption-ID Based Elimination For Multiple Choice Questions
Multiple choice questions (MCQs) are a popular and important task for evaluating large language models (LLMs). Based on common strategies people use when answering MCQs, the process of elimination (PoE) has been proposed…
Multiple-choice