Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop
We present our experience as annotators in the creation of high-quality, adversarial machine-reading-comprehension data for extractive QA for Task 1 of the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an emergent data collection paradigm with both models and humans in the loop. We set up a quasi-experimental annotation design and perform quantitative analyses across groups with different numbers of annotators focusing on successful adversarial attacks, cost analysis, and annotator confidence correlation. We further perform a qualitative analysis of our perceived difficulty of the task given the different topics of the passages in our dataset and conclude with recommendations and suggestions that might be of value to people working on future DADC tasks and related annotation interfaces.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine Reading ComprehensionReading ComprehensionSimilar Papers 제목 키워드 기반
Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data
Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. This study presents a comprehensive inves…
Machine TranslationTranslationMachine Learning for Windows Malware Detection and Classification: Methods, Challenges and Ongoing Research
In this chapter, readers will explore how machine learning has been applied to build malware detection systems designed for the Windows operating system. This chapter starts by introducing the main components of a Machin…
Malware DetectionAdversarial Robustness in Unsupervised Machine Learning: A Systematic Review
As the adoption of machine learning models increases, ensuring robust models against adversarial attacks is increasingly important. With unsupervised machine learning gaining more attention, ensuring it is robust against…
Adversarial RobustnessSystematic Literature ReviewA Reinforced Generation of Adversarial Examples for Neural Machine Translation
Neural machine translation systems tend to fail on less decent inputs despite its significant efficacy, which may significantly harm the credibility of this systems-fathoming how and when neural-based systems fail in suc…
Machine TranslationReinforcement LearningTranslationCOVID-19 CT Image Synthesis with a Conditional Generative Adversarial Network
Coronavirus disease 2019 (COVID-19) is an ongoing global pandemic that has spread rapidly since December 2019. Real-time reverse transcription polymerase chain reaction (rRT-PCR) and chest computed tomography (CT) imagin…
Computed Tomography (CT)COVID-19 DiagnosisDeep LearningGenerative Adversarial Network+2