Revisiting Distance Metric Learning for Few-Shot Natural Language Classification

2022-11-28 · Witold Sosnowski, Anna Wróblewska, Karolina Seweryn, Piotr Gawrysiak

Distance Metric Learning (DML) has attracted much attention in image processing in recent years. This paper analyzes its impact on supervised fine-tuning language models for Natural Language Processing (NLP) classification tasks under few-shot learning settings. We investigated several DML loss functions in training RoBERTa language models on known SentEval Transfer Tasks datasets. We also analyzed the possibility of using proxy-based DML losses during model inference. Our systematic experiments have shown that under few-shot learning settings, particularly proxy-based DML losses can positively affect the fine-tuning and inference of a supervised language model. Models tuned with a combination of CCE (categorical cross-entropy loss) and ProxyAnchor Loss have, on average, the best performance and outperform models with only CCE by about 3.27 percentage points -- up to 10.38 percentage points depending on the training dataset.

📄 PDF Abstract BibTeX arXiv:2211.15202

Code (0)

등록된 구현이 없습니다.

Tasks

Few-Shot LearningLanguage ModelingLanguage Modelling Metric Learning

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Weight Decay 설명 없음

Residual Connection 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

WordPiece 설명 없음

Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance

2024-04-12 · Yewei Song, Cedric Lothritz, Daniel Tang, Tegawendé F. Bissyandé 외

This paper revisits recent code similarity evaluation metrics, particularly focusing on the application of Abstract Syntax Tree (AST) editing distance in diverse programming languages. In particular, we explore the usefu…

Deep Metric Learning to Rank

2019-06-01 · CVPR 2019 6 · Fatih Cakir, Kun He, Xide Xia, Brian Kulis 외

We propose a novel deep metric learning method by revisiting the learning to rank approach. Our method, named FastAP, optimizes the rank-based Average Precision measure, using an approximation derived from distance quant…

Image Retrieval Learning-To-Rank Metric Learning Quantization+1

Revisiting Metric Learning for Few-Shot Image Classification

2019-07-06 · Xiaomeng Li, Lequan Yu, Chi-Wing Fu, Meng Fang 외

The goal of few-shot learning is to recognize new visual concepts with just a few amount of labeled samples in each class. Recent effective metric-based few-shot approaches employ neural networks to learn a feature simil…

Classification Few-Shot Image Classification Few-Shot Learning General Classification+5

One-Shot Learning for Language Modelling

2020-07-19 · Talip Ucar, Adrian Gonzalez-Martin, Matthew Lee, Adrian Daniel Szwarc

Humans can infer a great deal about the meaning of a word, using the syntax and semantics of surrounding words even if it is their first time reading or hearing it. We can also generalise the learned concept of the word …

BIG-bench Machine Learning Language Modelling One-Shot Learning

Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications

2023-10-21 · Manuel Faysse, Gautier Viaud, Céline Hudelot, Pierre Colombo

Instruction Fine-Tuning (IFT) is a powerful paradigm that strengthens the zero-shot capabilities of Large Language Models (LLMs), but in doing so induces new evaluation metric requirements. We show LLM-based metrics to b…