Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering
Ambiguity in natural language poses significant challenges to Large Language Models (LLMs) used for open-domain question answering. LLMs often struggle with the inherent uncertainties of human communication, leading to misinterpretations, miscommunications, hallucinations, and biased responses. This significantly weakens their ability to be used for tasks like fact-checking, question answering, feature extraction, and sentiment analysis. Using open-domain question answering as a test case, we compare off-the-shelf and few-shot LLM performance, focusing on measuring the impact of explicit disambiguation strategies. We demonstrate how simple, training-free, token-level disambiguation methods may be effectively used to improve LLM performance for ambiguous question answering tasks. We empirically show our findings and discuss best practices and broader impacts regarding ambiguity in LLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Fact CheckingOpen-Domain Question AnsweringQuestion AnsweringSentiment AnalysisSimilar Papers 제목 키워드 기반
Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity
In this work, we study a critical research problem regarding the trustworthiness of large language models (LLMs): how LLMs behave when encountering ambiguous narrative text, with a particular focus on Chinese textual amb…
AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries
Practical semantic parsers are expected to understand user utterances and map them to executable programs, even when these are ambiguous. We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the …
Text to SQLText-To-SQLLLMs as Deceptive Agents: How Role-Based Prompting Induces Semantic Ambiguity in Puzzle Tasks
Recent advancements in Large Language Models (LLMs) have not only showcased impressive creative capabilities but also revealed emerging agentic behaviors that exploit linguistic ambiguity in adversarial settings. In this…
Decision MakingFairnessText Is Not All You Need: Multimodal Prompting Helps LLMs Understand Humor
While Large Language Models (LLMs) have demonstrated impressive natural language understanding capabilities across various text-based tasks, understanding humor has remained a persistent challenge. Humor is frequently mu…
AllNatural Language UnderstandingRhythmtext-to-speech+1Reasoning about Ambiguous Definite Descriptions
Natural language reasoning plays an increasingly important role in improving language models' ability to solve complex language understanding tasks. An interesting use case for reasoning is the resolution of context-depe…