TIMEDIAL: Temporal Commonsense Reasoning in Dialog

2021-06-08 · ACL 2021 5 · Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, Manaal Faruqui

Everyday conversations require understanding everyday events, which in turn, requires understanding temporal commonsense concepts interwoven with those events. Despite recent progress with massive pre-trained language models (LMs) such as T5 and GPT-3, their capability of temporal reasoning in dialogs remains largely under-explored. In this paper, we present the first study to investigate pre-trained LMs for their temporal reasoning capabilities in dialogs by introducing a new task and a crowd-sourced English challenge set, TIMEDIAL. We formulate TIME-DIAL as a multiple-choice cloze task with over 1.1K carefully curated dialogs. Empirical results demonstrate that even the best performing models struggle on this task compared to humans, with 23 absolute points of gap in accuracy. Furthermore, our analysis reveals that the models fail to reason about dialog context correctly; instead, they rely on shallow cues based on existing temporal patterns in context, motivating future research for modeling temporal concepts in text and robust contextual reasoning about them. The dataset is publicly available at: https://github.com/google-research-datasets/timedial.

📄 PDF Abstract BibTeX arXiv:2106.04571

Code (1)

google-research-datasets/timedial 공식 구현

Tasks

Multiple-choiceTimedial

Methods 이 논문이 사용한 방법론

15 Ways to Contact How can i speak to someone at Delta Airlines 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Weight Decay 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

Adam 설명 없음

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

CIDER: Commonsense Inference for Dialogue Explanation and Reasoning

2021-06-01 · SIGDIAL (ACL) 2021 7 · Deepanway Ghosal, Pengfei Hong, Siqi Shen, Navonil Majumder 외

Commonsense inference to understand and explain human language is a fundamental research problem in natural language processing. Explaining human conversations poses a great challenge as it requires contextual understand…

Natural Language Inference

Leveraging Explicit Reasoning for Inference Integration in Commonsense-Augmented Dialogue Models

2024-06-13 · Sarah E. Finch, Jinho D. Choi

Open-domain dialogue systems need to grasp social commonsense to understand and respond effectively to human users. Commonsense-augmented dialogue models have been proposed that aim to infer commonsense knowledge from di…

Response Generation Specificity

ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems

2023-05-12 · Sarik Ghazarian, Yijia Shao, Rujun Han, Aram Galstyan 외

Commonsense reasoning is omnipresent in human communications and thus is an important feature for open-domain dialogue systems. However, evaluating commonsense in dialogue systems is still an open challenge. We take the …

CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models

2023-12-20 · Dan Shi, Chaobin You, Jiantao Huang, Taihao Li 외

As an indispensable ingredient of intelligence, commonsense reasoning is crucial for large language models (LLMs) in real-world scenarios. In this paper, we propose CORECODE, a dataset that contains abundant commonsense …

Causal Inference Common Sense Reasoning

CICERO: A Dataset for Contextualized Commonsense Inference in Dialogues

2022-03-25 · ACL 2022 5 · Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea 외

This paper addresses the problem of dialogue reasoning with contextualized commonsense inference. We curate CICERO, a dataset of dyadic conversations with five types of utterance-level reasoning-based inferences: cause, …

Answer Generation Answer Selection