AutoTinyBERT

2000년 도입 · 논문 2편에서 사용

AutoTinyBERT is a an efficient BERT variant found through neural architecture search. Specifically, one-shot learning is used to obtain a big Super Pretrained Language Model (SuperPLM), where the objectives of pre-training or task-agnostic BERT distillation are used. Then, given a specific latency constraint, an evolutionary algorithm is run on the SuperPLM to search optimal architectures. Finally, we extract the corresponding sub-models based on the optimal architectures and further train these models.

출처: AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models

소개 논문: AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models