Revisiting Few-sample BERT Fine-tuning
This paper is a study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process. In light of these observations, we re-visit recently proposed methods to improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe the impact of these methods diminishes significantly with our modified process.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Revisiting the Role of Label Smoothing in Enhanced Text Sentiment Classification
Label smoothing is a widely used technique in various domains, such as text classification, image classification and speech recognition, known for effectively combating model overfitting. However, there is little fine-gr…
Classificationimage-classificationImage ClassificationSentiment Analysis+5Parameter-Efficient Transfer Learning for NLP
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As …
Image ClassificationText ClassificationTransfer LearningMemorization of Named Entities in Fine-tuned BERT Models
Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data extraction from language models that ha…
MemorizationPrivacy PreservingPrivacy Preserving Deep Learningtext-classification+2Zero-Shot Prompting and Few-Shot Fine-Tuning: Revisiting Document Image Classification Using Large Language Models
Classifying scanned documents is a challenging problem that involves image, layout, and text analysis for document understanding. Nevertheless, for certain benchmark datasets, notably RVL-CDIP, the state of the art is cl…
Document Classificationdocument-image-classificationDocument Image Classificationdocument understanding+2Revisiting Distance Metric Learning for Few-Shot Natural Language Classification
Distance Metric Learning (DML) has attracted much attention in image processing in recent years. This paper analyzes its impact on supervised fine-tuning language models for Natural Language Processing (NLP) classificati…
Few-Shot LearningLanguage ModelingLanguage ModellingMetric Learning