Early Exiting BERT for Efficient Document Ranking
Pre-trained language models such as BERT have shown their effectiveness in various tasks. Despite their power, they are known to be computationally intensive, which hinders real-world applications. In this paper, we introduce early exiting BERT for document ranking. With a slight modification, BERT becomes a model with multiple output paths, and each inference sample can exit early from these paths. In this way, computation can be effectively allocated among samples, and overall system latency is significantly reduced while the original quality is maintained. Our experiments on two document ranking datasets demonstrate up to 2.5x inference speedup with minimal quality degradation. The source code of our implementation can be found at https://github.com/castorini/earlyexiting-monobert.
Code (1)
Tasks
Document RankingSimilar Papers 제목 키워드 기반
Query-level Early Exit for Additive Learning-to-Rank Ensembles
Search engine ranking pipelines are commonly based on large ensembles of machine-learned decision trees. The tight constraints on query response time recently motivated researchers to investigate algorithms to make faste…
Learning-To-RankVEE-BERT: Accelerating BERT Inference for Named Entity Recognition via Vote Early Exiting
Named entity recognition (NER) is of great importance for a wide range of tasks, such as medical health record understanding, document analysis, dialogue understanding. BERT and its variants are the most performing model…
Dialogue Understandingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+1SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
Dynamic early exiting has been proven to improve the inference speed of the pre-trained language model like BERT. However, all samples must go through all consecutive layers before early exiting and more complex samples …
Contrastive LearningLanguage ModellingRTEWNLIBERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression
The slow speed of BERT has motivated much research on accelerating its inference, and the early exiting idea has been proposed to make trade-offs between model quality and efficiency. This paper aims to address two weakn…
regressionDE$^3$-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks
Early exiting has demonstrated its effectiveness in accelerating the inference of pre-trained language models like BERT by dynamically adjusting the number of layers executed. However, most existing early exiting methods…