Uncertainty-aware Self-training for Text Classification with Few Labels

2020-06-27 · NeurIPS 2020 12 · Subhabrata Mukherjee, Ahmed Hassan Awadallah

Recent success of large-scale pre-trained language models crucially hinge on fine-tuning them on large amounts of labeled data for the downstream task, that are typically expensive to acquire. In this work, we study self-training as one of the earliest semi-supervised learning approaches to reduce the annotation bottleneck by making use of large-scale unlabeled data for the target task. Standard self-training mechanism randomly samples instances from the unlabeled pool to pseudo-label and augment labeled data. In this work, we propose an approach to improve self-training by incorporating uncertainty estimates of the underlying neural network leveraging recent advances in Bayesian deep learning. Specifically, we propose (i) acquisition functions to select instances from the unlabeled pool leveraging Monte Carlo (MC) Dropout, and (ii) learning mechanism leveraging model confidence for self-training. As an application, we focus on text classification on five benchmark datasets. We show our methods leveraging only 20-30 labeled samples per class for each task for training and for validation can perform within 3% of fully supervised pre-trained language models fine-tuned on thousands of labeled instances with an aggregate accuracy of 91% and improving by upto 12% over baselines.

📄 PDF Abstract BibTeX arXiv:2006.15315

Code (0)

등록된 구현이 없습니다.

Tasks

General ClassificationPseudo Labeltext-classification Text Classification

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

AcTune: Uncertainty-Aware Active Self-Training for Active Fine-Tuning of Pretrained Language Models

2022-01-16 · ACL ARR January 2022 1 · Anonymous

Although fine-tuning pre-trained language models (PLMs) renders strong performance in many NLP tasks, it relies on excessive labeled data. Recently, researchers have resorted to active fine-tuning for enhancing the label…

Active Learning text-classification Text Classification

Uncertainty-Aware Graph Self-Training with Expectation-Maximization Regularization

2025-03-26 · Emily Wang, Michael Chen, Chao Li

In this paper, we propose a novel \emph{uncertainty-aware graph self-training} approach for semi-supervised node classification. Our method introduces an Expectation-Maximization (EM) regularization scheme to incorporate…

Node Classification Pseudo Label

AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language Models

2021-12-16 · Yue Yu, Lingkai Kong, Jieyu Zhang, Rongzhi Zhang 외

While pre-trained language model (PLM) fine-tuning has achieved strong performance in many NLP tasks, the fine-tuning stage can be still demanding in labeled data. Recent works have resorted to active fine-tuning to impr…

Active Learning Language Modeling Language Modelling text-classification+1

Uncertainty-aware self-training with expectation maximization basis transformation

2024-05-02 · Zijia Wang, Wenbin Yang, ZhiSong Liu, Zhen Jia

Self-training is a powerful approach to deep learning. The key process is to find a pseudo-label for modeling. However, previous self-training algorithms suffer from the over-confidence issue brought by the hard labels, …

image-classification Image Classification Pseudo Label Semantic Segmentation

Graph-Based Uncertainty-Aware Self-Training with Stochastic Node Labeling

2025-03-26 · Tom Liu, Anna Wu, Chao Li

Self-training has become a popular semi-supervised learning technique for leveraging unlabeled data. However, the over-confidence of pseudo-labels remains a key challenge. In this paper, we propose a novel \emph{graph-ba…