Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models

2023-05-02 · Junmo Kang, Wei Xu, Alan Ritter

Fine-tuning large models is highly effective, however, inference can be expensive and produces carbon emissions. Knowledge distillation has been shown to be a practical solution to reduce inference costs, but the distillation process itself requires significant computational resources. Rather than buying or renting GPUs to fine-tune, then distill a large model, an NLP practitioner might instead choose to allocate the available budget to hire annotators and manually label additional fine-tuning data. In this paper, we investigate how to most efficiently use a fixed budget to build a compact model. Through extensive experiments on six diverse tasks, we show that distilling from T5-XXL (11B) to T5-Small (60M) is almost always a cost-efficient strategy compared to annotating more data to directly train a compact model (T5-Small). We further investigate how the optimal budget allocated towards computation varies across scenarios. We will make our code, datasets, annotation cost estimates, and baseline models available as a benchmark to support further work on cost-efficient training of compact models.

📄 PDF Abstract BibTeX arXiv:2305.01645

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT

2023-05-22 · Yichong Xu, Ruochen Xu, Dan Iter, Yang Liu 외

While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications. Conversely…

KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation

2024-10-28 · Rambod Azimi, Rishav Rishav, Marek Teichmann, Samira Ebrahimi Kahou

Large language models (LLMs) have demonstrated remarkable performance across various downstream tasks. However, the high computational and memory requirements of LLMs are a major bottleneck. To address this, parameter-ef…

GPU Knowledge Distillation parameter-efficient fine-tuning

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

2019-08-23 · ICLR 2020 1 · Iulia Turc, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of apply…

Knowledge Distillation Language Modelling Model Compression Sentiment Analysis

Hierarchical Prompt-Domain Control and Learning for Resource-Constrained Agentic Language Models

2026-05-26 · Joan Vendrell Gallart, Russell Bent, Michael Grosskopf arxiv

Large Language Models are increasingly deployed inside agentic systems, where they must follow structured protocols, adapt to evolving states, and operate under memory, latency, and cost constraints. In such regimes, pro…

Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelers

2026-01-06 · Yue Kang, Zhuoyi Huang, Benji Schussheim, Diana Licon 외 arxiv

In enterprise search, building high-quality datasets at scale remains a central challenge due to the difficulty of acquiring labeled data. To resolve this challenge, we propose an efficient approach to fine-tune small la…

Synthetic Data Generation