Multi-Strategy Knowledge Distillation Based Teacher-Student Framework for Machine Reading Comprehension
“The irrelevant information in documents poses a great challenge for machine reading compre-hension (MRC). To deal with such a challenge current MRC models generally fall into twoseparate parts: evidence extraction and answer prediction where the former extracts the key evi-dence corresponding to the question and the latter predicts the answer based on those sentences.However such pipeline paradigms tend to accumulate errors i.e. extracting the incorrect evi-dence results in predicting the wrong answer. In order to address this problem we propose aMulti-Strategy Knowledge Distillation based Teacher-Student framework (MSKDTS) for ma-chine reading comprehension. In our approach we first take evidence and document respec-tively as the input reference information to build a teacher model and a student model. Then the multi-strategy knowledge distillation method transfers the knowledge from the teacher model to the student model at both feature and prediction level through knowledge distillation approach.Therefore in the testing phase the enhanced student model can predict answer similar to the teacher model without being aware of which sentence is the corresponding evidence in the docu-ment. Experimental results on the ReCO dataset demonstrate the effectiveness of our approachand further ablation studies prove the effectiveness of both knowledge distillation strategies.”
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationMachine Reading ComprehensionReading ComprehensionSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Interactive Knowledge Distillation
Knowledge distillation is a standard teacher-student learning framework to train a light-weight student network under the guidance of a well-trained large teacher network. As an effective teaching strategy, interactive t…
image-classificationImage ClassificationKnowledge DistillationChannel Distillation: Channel-Wise Attention for Knowledge Distillation
Knowledge distillation is to transfer the knowledge from the data learned by the teacher network to the student network, so that the student has the advantage of less parameters and less calculations, and the accuracy is…
Knowledge DistillationAugmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression
Knowledge distillation (KD) is an effective model compression technique where a compact student network is taught to mimic the behavior of a complex and highly trained teacher network. In contrast, Mutual Learning (ML) p…
Knowledge DistillationModel Compressionobject-detectionObject Detection+1Collaborative Teacher-Student Learning via Multiple Knowledge Transfer
Knowledge distillation (KD), as an efficient and effective model compression technique, has been receiving considerable attention in deep learning. The key to its success is to transfer knowledge from a large teacher net…
Knowledge DistillationModel CompressionSelf-LearningTransfer LearningAdaptive Multi-Teacher Multi-level Knowledge Distillation
Knowledge distillation~(KD) is an effective learning paradigm for improving the performance of lightweight student networks by utilizing additional supervision knowledge distilled from teacher networks. Most pioneering s…
Knowledge Distillation