VisualMRC: Machine Reading Comprehension on Document Images
Recent studies on machine reading comprehension have focused on text-level understanding but have not yet reached the level of human understanding of the visual layout and content of real-world documents. In this study, we introduce a new visual machine reading comprehension dataset, named VisualMRC, wherein given a question and a document image, a machine reads and comprehends texts in the image to answer the question in natural language. Compared with existing visual question answering (VQA) datasets that contain texts in images, VisualMRC focuses more on developing natural language understanding and generation abilities. It contains 30,000+ pairs of a question and an abstractive answer for 10,000+ document images sourced from multiple domains of webpages. We also introduce a new model that extends existing sequence-to-sequence models, pre-trained with large-scale text corpora, to take into account the visual layout and content of documents. Experiments with VisualMRC show that this model outperformed the base sequence-to-sequence models and a state-of-the-art VQA model. However, its performance is still below that of humans on most automatic evaluation metrics. The dataset will facilitate research aimed at connecting vision and language understanding.
Code (1)
Tasks
Machine Reading ComprehensionNatural Language UnderstandingQuestion AnsweringReading ComprehensionVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
RoR: Read-over-Read for Long Document Machine Reading Comprehension
Transformer-based pre-trained models, such as BERT, have achieved remarkable results on machine reading comprehension. However, due to the constraint of encoding length (e.g., 512 WordPiece tokens), a long document is us…
Machine Reading ComprehensionReading ComprehensionTriviaQAMulti Document Reading Comprehension
Reading Comprehension (RC) is a task of answering a question from a given passage or a set of passages. In the case of multiple passages, the task is to find the best possible answer to the question. Recent trials and ex…
Question AnsweringReading ComprehensionCJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension
We present a Chinese judicial reading comprehension (CJRC) dataset which contains approximately 10K documents and almost 50K questions with answers. The documents come from judgment documents and the questions are annota…
DiversityMachine Reading ComprehensionReading ComprehensionA Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension
Reading Comprehension (RC) of text is one of the fundamental tasks in natural language processing. In recent years, several end-to-end neural network models have been proposed to solve RC tasks. However, most of these mo…
Machine Reading ComprehensionQuestion AnsweringReading ComprehensionTriviaQAA Reading Comprehension Corpus for Machine Translation Evaluation
Effectively assessing Natural Language Processing output tasks is a challenge for research in the area. In the case of Machine Translation (MT), automatic metrics are usually preferred over human evaluation, given time a…
Machine TranslationReading ComprehensionTranslation