TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing

2020-02-28 · ACL 2020 6 · Ziqing Yang, Yiming Cui, Zhipeng Chen, Wanxiang Che, Ting Liu, Shijin Wang, Guoping Hu

In this paper, we introduce TextBrewer, an open-source knowledge distillation toolkit designed for natural language processing. It works with different neural network models and supports various kinds of supervised learning tasks, such as text classification, reading comprehension, sequence labeling. TextBrewer provides a simple and uniform workflow that enables quick setting up of distillation experiments with highly flexible configurations. It offers a set of predefined distillation methods and can be extended with custom code. As a case study, we use TextBrewer to distill BERT on several typical NLP tasks. With simple configurations, we achieve results that are comparable with or even higher than the public distilled BERT models with similar numbers of parameters. Our toolkit is available through: http://textbrewer.hfl-rc.com

📄 PDF Abstract BibTeX arXiv:2002.12620

Code (1)

airaria/TextBrewer 공식 구현 pytorch

Tasks

Knowledge Distillation Reading Comprehension text-classification Text Classification

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Residual Connection 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Weight Decay 설명 없음

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models

2025-05-27 · Chengyu Wang, Junbing Yan, Wenrui Cai, Yuanhao Yue 외

In this paper, we present EasyDistill, a comprehensive toolkit designed for effective black-box and white-box knowledge distillation (KD) of large language models (LLMs). Our framework offers versatile functionalities, i…

Knowledge Distillation

N-LTP: An Open-source Neural Language Technology Platform for Chinese

2020-09-24 · EMNLP (ACL) 2021 11 · Wanxiang Che, Yunlong Feng, Libo Qin, Ting Liu

We introduce \texttt{N-LTP}, an open-source neural language technology platform supporting six fundamental Chinese NLP tasks: {lexical analysis} (Chinese word segmentation, part-of-speech tagging, and named entity recogn…

Chinese Word Segmentation Dependency Parsing Knowledge Distillation Lexical Analysis+7

LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis

2023-08-18 · Di Chang, Yufeng Yin, Zongjian Li, Minh Tran 외

Facial expression analysis is an important tool for human-computer interaction. In this paper, we introduce LibreFace, an open-source toolkit for facial expression analysis. This open-source toolbox offers real-time and …

Facial Expression Recognition Knowledge Distillation

DaNLP: An open-source toolkit for Danish Natural Language Processing

2021-05-01 · NoDaLiDa 2021 5 · Amalie Brogaard Pauli, Maria Barrett, Ophélie Lacroix, Rasmus Hvingelby

We present an open-source toolkit for Danish Natural Language Processing, enabling easy access to Danish NLP’s latest advancements. The toolkit features wrapper-functions for loading models and datasets in a unified way …

YANMTT: Yet Another Neural Machine Translation Toolkit

2021-08-25 · Raj Dabre, Eiichiro Sumita

In this paper we present our open-source neural machine translation (NMT) toolkit called "Yet Another Neural Machine Translation Toolkit" abbreviated as YANMTT which is built on top of the Transformers library. Despite t…

Machine Translation Model Compression NMT Transfer Learning+1