Alternating Training-based Label Smoothing Enhances Prompt Generalization
Recent advances in pre-trained vision-language models have demonstrated remarkable zero-shot generalization capabilities. To further enhance these models' adaptability to various downstream tasks, prompt tuning has emerged as a parameter-efficient fine-tuning method. However, despite its efficiency, the generalization ability of prompt remains limited. In contrast, label smoothing (LS) has been widely recognized as an effective regularization technique that prevents models from becoming over-confident and improves their generalization. This inspires us to explore the integration of LS with prompt tuning. However, we have observed that the vanilla LS even weakens the generalization ability of prompt tuning. To address this issue, we propose the Alternating Training-based Label Smoothing (ATLaS) method, which alternately trains with standard one-hot labels and soft labels generated by LS to supervise the prompt tuning. Moreover, we introduce two types of efficient offline soft labels, including Class-wise Soft Labels (CSL) and Instance-wise Soft Labels (ISL), to provide inter-class or instance-class relationships for prompt tuning. The theoretical properties of the proposed ATLaS method are analyzed. Extensive experiments demonstrate that the proposed ATLaS method, combined with CSL and ISL, consistently enhances the generalization performance of prompt tuning. Moreover, the proposed ATLaS method exhibits high compatibility with prevalent prompt tuning methods, enabling seamless integration into existing methods.
Code (0)
등록된 구현이 없습니다.
Tasks
parameter-efficient fine-tuningZero-shot GeneralizationSimilar Papers 제목 키워드 기반
Revisiting the Role of Label Smoothing in Enhanced Text Sentiment Classification
Label smoothing is a widely used technique in various domains, such as text classification, image classification and speech recognition, known for effectively combating model overfitting. However, there is little fine-gr…
Classificationimage-classificationImage ClassificationSentiment Analysis+5Instance-based Label Smoothing For Better Calibrated Classification Networks
Label smoothing is widely used in deep neural networks for multi-class classification. While it enhances model generalization and reduces overconfidence by aiming to lower the probability for the predicted class, it dist…
Classificationimage-classificationImage ClassificationMulti-class ClassificationStructure-Aware Label Smoothing for Graph Neural Networks
Representing a label distribution as a one-hot vector is a common practice in training node classification models. However, the one-hot representation may not adequately reflect the semantic characteristics of a node in …
ClassificationNode ClassificationImproving Predictive Confidence in Medical Imaging via Online Label Smoothing
Deep learning models, especially convolutional neural networks, have achieved impressive results in medical image classification. However, these models often produce overconfident predictions, which can undermine their r…
Medical Image ClassificationRepresentation LearningKnowledge DistillationFedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration
Deep Neural Network (DNN) classifiers suffer from poor calibration when their softmax outputs (predictive confidence) deviate from the empirical likelihoods. This manifests itself as either overconfident incorrect predic…