ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots
We introduce ScreenQA, a novel benchmarking dataset designed to advance screen content understanding through question answering. The existing screen datasets are focused either on low-level structural and component understanding, or on a much higher-level composite task such as navigation and task completion for autonomous agents. ScreenQA attempts to bridge this gap. By annotating 86k question-answer pairs over the RICO dataset, we aim to benchmark the screen reading comprehension capacity, thereby laying the foundation for vision-based automation over screenshots. Our annotations encompass full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. We evaluate the dataset's efficacy using both open-weight and proprietary models in zero-shot, fine-tuned, and transfer learning settings. We further demonstrate positive transfer to web applications, highlighting its potential beyond mobile applications.
Code (1)
Tasks
BenchmarkingQuestion AnsweringReading ComprehensionTransfer LearningSimilar Papers 제목 키워드 기반
Training IBM Watson using Automatically Generated Question-Answer Pairs
IBM Watson is a cognitive computing system capable of question answering in natural languages. It is believed that IBM Watson can understand large corpora and answer relevant questions more effectively than any other que…
Answer GenerationQuestion-Answer-GenerationQuestion AnsweringRole play-based question-answering by real users for building chatbots with consistent personalities
Having consistent personalities is important for chatbots if we want them to be believable. Typically, many question-answer pairs are prepared by hand for achieving consistent responses; however, the creation of such pai…
ChatbotQuestion AnsweringOneStop QAMaker: Extract Question-Answer Pairs from Text in a One-Stop Approach
Large-scale question-answer (QA) pairs are critical for advancing research areas like machine reading comprehension and question answering. To construct QA pairs from documents requires determining how to ask a question …
Machine Reading ComprehensionQuestion AnsweringQuestion GenerationQuestion-Generation+1CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
With the rise of large-scale pre-trained language models, open-domain question-answering (ODQA) has become an important research topic in NLP. Based on the popular pre-training fine-tuning approach, we posit that an addi…
Open-Domain Question AnsweringQuestion AnsweringA Unified Abstractive Model for Generating Question-Answer Pairs
Large-scale question-answer pairs (QAP) are valuable for many applications, such as knowledge bases construction and machine reading comprehension. Although its importance has been widely recognized, existing approaches …
Answer GenerationMachine Reading ComprehensionQuestion GenerationQuestion-Generation+1