SCDE: Sentence Cloze Dataset with High Quality Distractors From Examinations
We introduce SCDE, a dataset to evaluate the performance of computational models through sentence prediction. SCDE is a human-created sentence cloze dataset, collected from public school English examinations. Our task requires a model to fill up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. Experimental results demonstrate that this task requires the use of non-local, discourse-level context beyond the immediate sentence neighborhood. The blanks require joint solving and significantly impair each other's context. Furthermore, through ablations, we show that the distractors are of high quality and make the task more challenging. Our experiments show that there is a significant performance gap between advanced models (72%) and humans (87%), encouraging future models to bridge this gap.
Code (1)
Tasks
Question AnsweringSentenceVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
Medical Image Segmentation via Sparse Coding Decoder
Transformers have achieved significant success in medical image segmentation, owing to its capability to capture long-range dependencies. Previous works incorporate convolutional layers into the encoder module of transfo…
DecoderImage SegmentationMedical Image SegmentationSemantic SegmentationExploring Sentence Community for Document-Level Event Extraction
Document-level event extraction is critical to various natural language processing tasks for providing structured information. Existing approaches by sequential modeling neglect the complex logic structures for long text…
Community DetectionDocument-level Event ExtractionEvent ExtractionGraph Attention+1InDEX: Indonesian Idiom and Expression Dataset for Cloze Test
We propose InDEX, an Indonesian Idiom and Expression dataset for cloze test. The dataset contains 10438 unique sentences for 289 idioms and expressions for which we generate 15 different types of distractors, resulting i…
Cloze TestReading ComprehensionA Simple and Effective Approach to the Story Cloze Test
In the Story Cloze Test, a system is presented with a 4-sentence prompt to a story, and must determine which one of two potential endings is the 'right' ending to the story. Previous work has shown that ignoring the trai…
Cloze TestFeature EngineeringQuestion AnsweringSentenceDualReward: A Dynamic Reinforcement Learning Framework for Cloze Tests Distractor Generation
This paper introduces DualReward, a novel reinforcement learning framework for automatic distractor generation in cloze tests. Unlike conventional approaches that rely primarily on supervised learning or static generativ…
Reinforcement LearningDistractor GenerationCloze Test