Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning

2023-08-26 · Apoorv Dankar, Adeem Jassani, Kartikaeya Kumar

The use of large transformer-based models such as BERT, GPT, and T5 has led to significant advancements in natural language processing. However, these models are computationally expensive, necessitating model compression techniques that reduce their size and complexity while maintaining accuracy. This project investigates and applies knowledge distillation for BERT model compression, specifically focusing on the TinyBERT student model. We explore various techniques to improve knowledge distillation, including experimentation with loss functions, transformer layer mapping methods, and tuning the weights of attention and representation loss and evaluate our proposed techniques on a selection of downstream tasks from the GLUE benchmark. The goal of this work is to improve the efficiency and effectiveness of knowledge distillation, enabling the development of more efficient and accurate models for a range of natural language processing tasks.

📄 PDF Abstract BibTeX arXiv:2308.13958

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation Model Compression

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

Weight Decay 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

WordPiece 설명 없음

Discriminative Fine-Tuning Discriminative Fine-Tuning is a fine-tuning strategy that is used for ULMFiT type models. Instead of using the same learning rate…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Weight-Inherited Distillation for Task-Agnostic BERT Compression

2023-05-16 · Taiqiang Wu, Cheng Hou, Shanshan Lao, Jiayi Li 외

Knowledge Distillation (KD) is a predominant approach for BERT compression. Previous KD-based methods focus on designing extra alignment losses for the student model to mimic the behavior of the teacher model. These meth…

Knowledge Distillation

Improving Task-Agnostic BERT Distillation with Layer Mapping Search

2020-12-11 · Xiaoqi Jiao, Huating Chang, Yichun Yin, Lifeng Shang 외

Knowledge distillation (KD) which transfers the knowledge from a large teacher model to a small student model, has been widely used to compress the BERT model recently. Besides the supervision in the output in the origin…

Knowledge Distillation

Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models

2022-01-03 · Made Nindyatama Nityasya, Haryo Akbarianto Wibowo, Rendi Chevi, Radityo Eko Prasojo 외

We perform knowledge distillation (KD) benchmark from task-specific BERT-base teacher models to various student models: BiLSTM, CNN, BERT-Tiny, BERT-Mini, and BERT-Small. Our experiment involves 12 datasets grouped in tw…

CPU Data Augmentation Knowledge Distillation text-classification+2

ListBERT: Learning to Rank E-commerce products with Listwise BERT

2022-06-30 · Lakshya Kumar, Sagnik Sarkar

Efficient search is a critical component for an e-commerce platform with an innumerable number of products. Every day millions of users search for products pertaining to their needs. Thus, showing the relevant products o…

Knowledge Distillation Learning-To-Rank

An Empirical Study of Uniform-Architecture Knowledge Distillation in Document Ranking

2023-02-08 · Xubo Qin, Xiyuan Liu, Xiongfeng Zheng, Jie Liu 외

Although BERT-based ranking models have been commonly used in commercial search engines, they are usually time-consuming for online ranking tasks. Knowledge distillation, which aims at learning a smaller model with compa…

Document Ranking Knowledge Distillation