Dual Multi-head Co-attention for Multi-choice Reading Comprehension
Multi-choice Machine Reading Comprehension (MRC) requires model to decide the correct answer from a set of answer options when given a passage and a question. Thus in addition to a powerful Pre-trained Language Model (PrLM) as encoder, multi-choice MRC especially relies on a matching network design which is supposed to effectively capture the relationships among the triplet of passage, question and answers. While the newer and more powerful PrLMs have shown their mightiness even without the support from a matching network, we propose a new DUal Multi-head Co-Attention (DUMA) model, which is inspired by human's transposition thinking process solving the multi-choice MRC problem: respectively considering each other's focus from the standpoint of passage and question. The proposed DUMA has been shown effective and is capable of generally promoting PrLMs. Our proposed method is evaluated on two benchmark multi-choice MRC tasks, DREAM and RACE, showing that in terms of powerful PrLMs, DUMA can still boost the model to reach new state-of-the-art performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMachine Reading ComprehensionMulti-Choice MRCReading ComprehensionTripletSimilar Papers 제목 키워드 기반
Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
We study the dynamics of gradient flow for training a multi-head softmax attention model for in-context learning of multi-task linear regression. We establish the global convergence of gradient flow under suitable choice…
In-Context LearningDoes Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla
\emph{Circuit analysis} is a promising technique for understanding the internal mechanisms of language models. However, existing analyses are done in small models far from the state of the art. To address this, we presen…
Multiple-choiceQuestion AnsweringDUMA: Reading Comprehension with Transposition Thinking
Multi-choice Machine Reading Comprehension (MRC) requires model to decide the correct answer from a set of answer options when given a passage and a question. Thus in addition to a powerful Pre-trained Language Model (Pr…
Language ModelingLanguage ModellingMachine Reading ComprehensionMulti-Choice MRC+2PartialFormer: Modeling Part Instead of Whole for Machine Translation
The design choices in Transformer feed-forward neural networks have resulted in significant computational and parameter overhead. In this work, we emphasize the importance of hidden dimensions in designing lightweight FF…
Abstractive Text SummarizationMachine TranslationTranslationArchitecture, Not Scale: Circuit Localization in Large Language Models
Mechanistic interpretability assumes that circuit analysis becomes harder as models scale. We challenge this assumption by showing that the attention architecture matters more than parameter count. Studying three circuit…