Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios
The possible consequences for the same context may vary depending on the situation we refer to. However, current studies in natural language processing do not focus on situated commonsense reasoning under multiple possible scenarios. This study frames this task by asking multiple questions with the same set of possible endings as candidate answers, given a short story text. Our resulting dataset, Possible Stories, consists of more than 4.5K questions over 1.3K story texts in English. We discover that even current strong pretrained language models struggle to answer the questions consistently, highlighting that the highest accuracy in an unsupervised setting (60.2%) is far behind human accuracy (92.5%). Through a comparison with existing datasets, we observe that the questions in our dataset contain minimal annotation artifacts in the answer options. In addition, our dataset includes examples that require counterfactual reasoning, as well as those requiring readers' reactions and fictional information, suggesting that our dataset can serve as a challenging testbed for future studies on situated commonsense reasoning.
Code (1)
Tasks
counterfactualCounterfactual ReasoningSimilar Papers 제목 키워드 기반
SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning
Recently, commonsense reasoning in text generation has attracted much attention. Generative commonsense reasoning is the task that requires machines, given a group of keywords, to compose a single coherent sentence with …
SentenceText GenerationSOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge
Learning commonsense reasoning from visual contexts and scenes in real-world is a crucial step toward advanced artificial intelligence. However, existing video reasoning benchmarks are still inadequate since they were ma…
General KnowledgeKnowledge GraphsWorld KnowledgeRule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?
Building machines with commonsense has been a longstanding challenge in NLP due to the reporting bias of commonsense rules and the exposure bias of rule-based commonsense reasoning. In contrast, humans convey and pass do…
Modeling Tension in Stories via Commonsense Reasoning and Emotional Word Embeddings
Dramatic tension is crucial for generating interesting stories. This paper aims to model dramatic tension from a story text using neural commonsense-reasoning language models and emotional word embeddings. We also propos…
Word EmbeddingsStory Generation with Commonsense Knowledge Graphs and Axioms
Humans can understand stories, and the rich interactions between agents, locations, and events, seamlessly. However, state-of-the-art reasoning models struggle with understanding, completing, or explaining stories, often…
Common Sense ReasoningKnowledge GraphsStory Generation