홈 › Papers

GradMask: Effective Fine-tuning on Large-scale Pretrained Language Models via Gradient Masking

Pretrained language models have dominated a variety of NLP tasks. However, fine-tuning large pretrained models on downstream tasks tend to achieve degenerated and unstable results, especially when there are only a limited number of training examples available. Common ways to solve the problem are via regularization like Dropout or other methods designed specially for fine-tuning, which are mainly applied during forward propagation. Orthogonal to them, we propose GradMask, a simple regularization during backward propagation. GradMask masks part of gradients of model parameters to zero after gradients computation, which prevents the overfitting on downstream tasks, and enables the model to explore potentially better solutions. We also propose a layer-adaptive masking strategy to fully transfer knowledge from pretrained models. Experiments demonstrate that fine-tuning BERT-LARGE with GradMask improves 1.26 mean score in 4 smaller datasets in GLUE, and consistent improvements are also gained with other pretrained models. Besides, We empirically find fine-tuning with GradMask enhances learning rate robustness, i.e., achieves promising results at a wide range of learning rate from 10^{-5} to 10^{-4}.

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Similar Papers 제목 키워드 기반