Distance Metric Learning Loss Functions in Few-Shot Scenarios of Supervised Language Models Fine-Tuning
This paper presents an analysis regarding an influence of the Distance Metric Learning (DML) loss functions on the supervised fine-tuning of the language models for classification tasks. We experimented with known datasets from SentEval Transfer Tasks. Our experiments show that applying the DML loss function can increase performance on downstream classification tasks of RoBERTa-large models in few-shot scenarios. Models fine-tuned with the use of SoftTriple loss can achieve better results than models with a standard categorical cross-entropy loss function by about 2.89 percentage points from 0.04 to 13.48 percentage points depending on the training dataset. Additionally, we accomplished a comprehensive analysis with explainability techniques to assess the models' reliability and explain their results.
Code (0)
등록된 구현이 없습니다.
Tasks
Metric LearningSimilar Papers 제목 키워드 기반
Assessing two novel distance-based loss functions for few-shot image classification
Few-shot learning is a challenging area of research which aims to learn new concepts with only a few labeled samples of data. Recent works based on metric-learning approaches benefit from the meta-learning process in whi…
Few-Shot Image ClassificationFew-Shot Learningimage-classificationImage Classification+3SuSana Distancia is all you need: Enforcing class separability in metric learning via two novel distance-based loss functions for few-shot image classification
Few-shot learning is a challenging area of research that aims to learn new concepts with only a few labeled samples of data. Recent works based on metric-learning approaches leverage the meta-learning approach, which is …
AllFew-Shot Image ClassificationFew-Shot Learningimage-classification+4Constellation Loss: Improving the efficiency of deep metric learning loss functions for optimal embedding
Metric learning has become an attractive field for research on the latest years. Loss functions like contrastive loss, triplet loss or multi-class N-pair loss have made possible generating models capable of tackling comp…
Few-Shot LearningMetric LearningTripletRevisiting Distance Metric Learning for Few-Shot Natural Language Classification
Distance Metric Learning (DML) has attracted much attention in image processing in recent years. This paper analyzes its impact on supervised fine-tuning language models for Natural Language Processing (NLP) classificati…
Few-Shot LearningLanguage ModelingLanguage ModellingMetric LearningSpectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey
This is a tutorial and survey paper on metric learning. Algorithms are divided into spectral, probabilistic, and deep metric learning. We first start with the definition of distance metric, Mahalanobis distance, and gene…
Dimensionality ReductionMetric LearningSurveyTriplet