Large-scale Cloze Test Dataset Created by Teachers
Cloze tests are widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.
Code (0)
등록된 구현이 없습니다.
Tasks
Cloze TestLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Large-scale Cloze Test Dataset Designed by Teachers
Cloze test is widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-designed cloze test dataset CLOTH in which the questions were used in middl…
Cloze TestChID: A Large-scale Chinese IDiom Dataset for Cloze Test
Cloze-style reading comprehension in Chinese is still limited due to the lack of various corpora. In this paper we propose a large-scale Chinese cloze test dataset ChID, which studies the comprehension of idiom, a unique…
Cloze TestReading ComprehensionDualReward: A Dynamic Reinforcement Learning Framework for Cloze Tests Distractor Generation
This paper introduces DualReward, a novel reinforcement learning framework for automatic distractor generation in cloze tests. Unlike conventional approaches that rely primarily on supervised learning or static generativ…
Reinforcement LearningDistractor GenerationCloze TestNLP and Education: using semantic similarity to evaluate filled gaps in a large-scale Cloze test in the classroom
This study examines the applicability of the Cloze test, a widely used tool for assessing text comprehension proficiency, while highlighting its challenges in large-scale implementation. To address these limitations, an …
Cloze TestReading ComprehensionSemantic SimilaritySemantic Textual Similarity+1InDEX: Indonesian Idiom and Expression Dataset for Cloze Test
We propose InDEX, an Indonesian Idiom and Expression dataset for cloze test. The dataset contains 10438 unique sentences for 289 idioms and expressions for which we generate 15 different types of distractors, resulting i…
Cloze TestReading Comprehension