FAT ALBERT: Finding Answers in Large Texts using Semantic Similarity Attention Layer based on BERT
Machine based text comprehension has always been a significant research field in natural language processing. Once a full understanding of the text context and semantics is achieved, a deep learning model can be trained to solve a large subset of tasks, e.g. text summarization, classification and question answering. In this paper we focus on the question answering problem, specifically the multiple choice type of questions. We develop a model based on BERT, a state-of-the-art transformer network. Moreover, we alleviate the ability of BERT to support large text corpus by extracting the highest influence sentences through a semantic similarity model. Evaluations of our proposed model demonstrate that it outperforms the leading models in the MovieQA challenge and we are currently ranked first in the leader board with test accuracy of 87.79%. Finally, we discuss the model shortcomings and suggest possible improvements to overcome these limitations.
Code (1)
Tasks
Multiple-choiceQuestion AnsweringReading ComprehensionSemantic SimilaritySemantic Textual SimilarityText SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Transformer-based Language Models for Factoid Question Answering at BioASQ9b
In this work, we describe our experiments and participating systems in the BioASQ Task 9b Phase B challenge of biomedical question answering. We have focused on finding the ideal answers and investigated multi-task fine-…
Question AnsweringProcessing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer
Pre-trained Transformers currently dominate most NLP tasks. They impose, however, limits on the maximum input length (512 sub-words in BERT), which are too restrictive in the legal domain. Even sparse-attention models, s…
Document ClassificationDiscourse-Aware Semantic Self-Attention for Narrative Reading Comprehension
In this work, we propose to use linguistic annotations as a basis for a \textit{Discourse-Aware Semantic Self-Attention} encoder that we employ for reading comprehension on long narrative texts. We extract relations betw…
Reading ComprehensionSentenceALBERTI, a Multilingual Domain Specific Language Model for Poetry Analysis
The computational analysis of poetry is limited by the scarcity of tools to automatically analyze and scan poems. In a multilingual settings, the problem is exacerbated as scansion and rhyme systems only exist for indivi…
Language ModelingLanguage ModellingLarge Language ModelGenerating Questions and Multiple-Choice Answers using Semantic Analysis of Texts
We present a novel approach to automated question generation that improves upon prior work both from a technology perspective and from an assessment perspective. Our system is aimed at engaging language learners by gener…
coreference-resolutionCoreference ResolutionMultiple-choiceQuestion Generation+3