Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

2023-05-26 · Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, Yanai Elazar

Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning due to its simplicity and improved out-of-domain generalization, and because extensive evidence shows that fine-tuned models pick up on spurious correlations. Unfortunately, previous comparisons of the two approaches were done using models of different sizes. This raises the question of whether the observed weaker out-of-domain generalization of fine-tuned models is an inherent property of fine-tuning or a limitation of the experimental setup. In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B. Our results show that fine-tuned language models can in fact generalize well out-of-domain. We find that both approaches generalize similarly; they exhibit large variation and depend on properties such as model size and the number of examples, highlighting that robust task adaptation remains a challenge.

📄 PDF Abstract BibTeX arXiv:2305.16938

Code (1)

uds-lsv/llmft 공식 구현 pytorch

Tasks

Domain GeneralizationIn-Context Learning

Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models

2023-11-14 · Carlos Aguirre, Kuleen Sasse, Isabel Cachola, Mark Dredze

Recently, work in NLP has shifted to few-shot (in-context) learning, with large language models (LLMs) performing well across a range of tasks. However, while fairness evaluations have become a standard for supervised me…

Fairness Few-Shot Learning In-Context Learning

Group Fairness Meets the Black Box: Enabling Fair Algorithms on Closed LLMs via Post-Processing

2025-08-15 · Ruicheng Xian, Yuxuan Wan, Han Zhao arxiv

Instruction fine-tuned large language models (LLMs) enable a simple zero-shot or few-shot prompting paradigm, also known as in-context learning, for building prediction models. This convenience, combined with continued a…

MEAL: Stable and Active Learning for Few-Shot Prompting

2022-11-15 · Abdullatif Köksal, Timo Schick, Hinrich Schütze

Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners. However, this approach has high variance both across different sets of …

Active Learning Few-Shot Learning RTE

Exploring Fine-Tuning for Tabular Foundation Models

2026-01-14 · Aditya Tanna, Pratinav Seth, Mohamed Bouadi, Vinay Kumar Sankarapu arxiv

Tabular Foundation Models (TFMs) have recently shown strong in-context learning capabilities on structured data, achieving zero-shot performance comparable to traditional machine learning methods. We find that zero-shot …

Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification

2024-02-28 · Garima Chhikara, Anurag Sharma, Kripabandhu Ghosh, Abhijnan Chakraborty

Employing Large Language Models (LLM) in various downstream applications such as classification is crucial, especially for smaller companies lacking the expertise and resources required for fine-tuning a model. Fairness …

Fairness In-Context Learning RAG