CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
Fine-tuning pre-trained language models (PLMs) has demonstrated its effectiveness on various downstream NLP tasks recently. However, in many low-resource scenarios, the conventional fine-tuning strategies cannot sufficiently capture the important semantic features for downstream tasks. To address this issue, we introduce a novel framework (named "CSS-LM") to improve the fine-tuning phase of PLMs via contrastive semi-supervised learning. Specifically, given a specific task, we retrieve positive and negative instances from large-scale unlabeled corpora according to their domain-level and class-level semantic relatedness to the task. We then perform contrastive semi-supervised learning on both the retrieved unlabeled and original labeled instances to help PLMs capture crucial task-related semantic features. The experimental results show that CSS-LM achieves better results than the conventional fine-tuning strategy on a series of downstream tasks with few-shot settings, and outperforms the latest supervised contrastive fine-tuning strategies. Our datasets and source code will be available to provide more details.
Code (1)
Similar Papers 제목 키워드 기반
Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss
Sound event detection (SED) is a core module for acoustic environmental analysis, yet its performance is often limited by scarce labeled data. Recent systems leverage large pretrained audio foundation models, but effecti…
Sound Event DetectionContrastive LearningSemi-Supervised End-To-End Contrastive Learning For Time Series Classification
Time series classification is a critical task in various domains, such as finance, healthcare, and sensor data analysis. Unsupervised contrastive learning has garnered significant interest in learning effective represent…
ClassificationContrastive LearningTime SeriesTime Series Classification+1ContraCluster: Learning to Classify without Labels by Contrastive Self-Supervision and Prototype-Based Semi-Supervision
The recent advances in representation learning inspire us to take on the challenging problem of unsupervised image classification tasks in a principled way. We propose ContraCluster, an unsupervised image classification …
Contrastive Learningimage-classificationImage ClassificationRepresentation Learning+2Differentiable Data Augmentation for Contrastive Sentence Representation Learning
Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sentence pairs is a common way to obtain high-quality sentence representations. Althou…
Contrastive LearningData AugmentationLanguage ModelingLanguage Modelling+2SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning
This paper introduces SelfMatch, a semi-supervised learning method that combines the power of contrastive self-supervised learning and consistency regularization. SelfMatch consists of two stages: (1) self-supervised pre…
Contrastive LearningSelf-Supervised LearningSemi-Supervised Image Classification