Automated Assessment of Noisy Crowdsourced Free-text Answers for Hindi in Low Resource Setting
The requirement of performing assessments continually on a larger scale necessitates the implementation of automated systems for evaluation of the learners’ responses to free-text questions. We target children of age group 8-14 years and use an ASR integrated assessment app to crowdsource learners’ responses to free text questions in Hindi. The app helped collect 39641 user answers to 35 different questions of Science topics. Since the users are young children from rural India and may not be well-equipped with technology, it brings in various noise types in the answers. We describe these noise types and propose a preprocessing pipeline to denoise user’s answers. We showcase the performance of different similarity metrics on the noisy and denoised versions of user and model answers. Our findings have large-scale applications for automated answer assessment for school children in India in low resource settings.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CaRB: A Crowdsourced Benchmark for Open IE
Open Information Extraction (Open IE) systems have been traditionally evaluated via manual annotation. Recently, an automated evaluator with a benchmark dataset (OIE2016) was released {--} it scores Open IE systems autom…
Open Information ExtractionV-RoAst: Visual Road Assessment. Can VLM be a Road Safety Assessor Using the iRAP Standard?
Road traffic crashes result in millions of deaths annually and significant economic burdens, particularly on Low- and Middle-Income Countries (LMICs). Road safety assessments traditionally rely on human-labelled data, wh…
Few-Shot LearningIn-Context Learningparameter-efficient fine-tuningPrompt Engineering+3Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Images
Automated detection of electron dense deposits (EDD) in glomerular disease is hindered by the scarcity of high-quality labeled data. While crowdsourcing reduces annotation cost, it introduces label noise. We propose an a…
Active LearningA Machine Learning Approach to Persian Text Readability Assessment Using a Crowdsourced Dataset
An automated approach to text readability assessment is essential to a language and can be a powerful tool for improving the understandability of texts written and published in that language. However, the Persian languag…
BIG-bench Machine LearningCrowdsourcing and Evaluating Text-Based Audio Retrieval Relevances
This paper explores grading text-based audio retrieval relevances with crowdsourcing assessments. Given a free-form text (e.g., a caption) as a query, crowdworkers are asked to grade audio clips using numeric scores (bet…
Audio captioningContrastive LearningRetrieval