SemEval-2018 Task 12: The Argument Reasoning Comprehension Task

2018-06-01 · SEMEVAL 2018 6 · Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, Benno Stein

A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal was to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of about 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Common Sense Reasoning

ITNLP-ARC at SemEval-2018 Task 12: Argument Reasoning Comprehension with Attention

2018-06-01 · SEMEVAL 2018 6 · Wenjie Liu, Chengjie Sun, Lei Lin, Bingquan Liu

Reasoning is a very important topic and has many important applications in the field of natural language processing. Semantic Evaluation (SemEval) 2018 Task 12 {``}The Argument Reasoning Comprehension{''} committed to re…

ARC Information Retrieval Machine Translation Natural Language Inference+4

YNU Deep at SemEval-2018 Task 12: A BiLSTM Model with Neural Attention for Argument Reasoning Comprehension

2018-06-01 · SEMEVAL 2018 6 · Peng Ding, Xiaobing Zhou

This paper describes the system submitted to SemEval-2018 Task 12 (The Argument Reasoning Comprehension Task). Enabling a computer to understand a text so that it can answer comprehension questions is still a challenging…

Constituency Parsing Language Modeling Language Modelling Machine Translation+1

Joker at SemEval-2018 Task 12: The Argument Reasoning Comprehension with Neural Attention

2018-06-01 · SEMEVAL 2018 6 · Guobin Sui, WenHan Chao, Zhunchen Luo

This paper describes a classification system that participated in the SemEval-2018 Task 12: The Argument Reasoning Comprehension Task. Briefly the task can be described as that a natural language {``}argument{''} is what…

Argument Mining Stance Detection

TakeLab at SemEval-2018 Task12: Argument Reasoning Comprehension with Skip-Thought Vectors

2018-06-01 · SEMEVAL 2018 6 · Ana Brassard, Tin Kuculo, Filip Boltu{\v{z}}i{\'c}, Jan {\v{S}}najder

This paper describes our system for the SemEval-2018 Task 12: Argument Reasoning Comprehension Task. We utilize skip-thought vectors, sentence-level distributional vectors inspired by the popular word embeddings and the …

Common Sense Reasoning Natural Language Inference Sentence Word Embeddings

ECNU at SemEval-2018 Task 12: An End-to-End Attention-based Neural Network for the Argument Reasoning Comprehension Task

2018-06-01 · SEMEVAL 2018 6 · Junfeng Tian, Man Lan, Yuanbin Wu

This paper presents our submissions to SemEval 2018 Task 12: the Argument Reasoning Comprehension Task. We investigate an end-to-end attention-based neural network to represent the two lexically close candidate warrants.…