Simplified TinyBERT: Knowledge Distillation for Document Retrieval
Despite the effectiveness of utilizing the BERT model for document ranking, the high computational cost of such approaches limits their uses. To this end, this paper first empirically investigates the effectiveness of two knowledge distillation models on the document ranking task. In addition, on top of the recently proposed TinyBERT model, two simplifications are proposed. Evaluations on two different and widely-used benchmarks demonstrate that Simplified TinyBERT with the proposed simplifications not only boosts TinyBERT, but also significantly outperforms BERT-Base when providing 15$\times$ speedup.
Code (4)
Tasks
Document RankingKnowledge DistillationRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
TinyBERT: Distilling BERT for Natural Language Understanding
Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is diffi…
Knowledge DistillationLanguage ModellingLinguistic AcceptabilityNatural Language Inference+5Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments
This study proposes a method for knowledge distillation (KD) of fine-tuned Large Language Models (LLMs) into smaller, more efficient, and accurate neural networks. We specifically target the challenge of deploying these …
Knowledge DistillationMathematical ReasoningFine-Grained Distillation for Long Document Retrieval
Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encode…
Knowledge DistillationRetrievalQuaLA-MiniLM: a Quantized Length Adaptive MiniLM
Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. A knowledge distillation approach addresses the computational efficiency by self-distil…
Computational EfficiencyKnowledge DistillationQuantizationQuestion AnsweringPROD: Progressive Distillation for Dense Retrieval
Knowledge distillation is an effective way to transfer knowledge from a strong teacher to an efficient student model. Ideally, we expect the better the teacher is, the better the student. However, this expectation does n…
Knowledge DistillationNatural QuestionsRetrieval