ClusterUCB: Efficient Gradient-Based Data Selection for Targeted Fine-Tuning of LLMs
Gradient-based data influence approximation has been leveraged to select useful data samples in the supervised fine-tuning of large language models. However, the computation of gradients throughout the fine-tuning process requires too many resources to be feasible in practice. In this paper, we propose an efficient gradient-based data selection framework with clustering and a modified Upper Confidence Bound (UCB) algorithm. Based on the intuition that data samples with similar gradient features will have similar influences, we first perform clustering on the training data pool. Then, we frame the inter-cluster data selection as a constrained computing budget allocation problem and consider it a multi-armed bandit problem. A modified UCB algorithm is leveraged to solve this problem. Specifically, during the iterative sampling process, historical data influence information is recorded to directly estimate the distributions of each cluster, and a cold start is adopted to balance exploration and exploitation. Experimental results on various benchmarks show that our proposed framework, ClusterUCB, can achieve comparable results to the original gradient-based data selection methods while greatly reducing computing consumption.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringInfluence ApproximationSimilar Papers 제목 키워드 기반
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, …
GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry
Targeted data selection has emerged as a crucial paradigm for efficient instruction tuning, aiming to identify a small yet influential subset of training examples for a specific target task. In practice, influence is oft…
parameter-efficient fine-tuningScalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach
We study the problem of fine-tuning a language model (LM) for a target task by optimally using the information from $n$ auxiliary tasks. This problem has broad applications in NLP, such as targeted instruction tuning and…
EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptation
Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge. Such adaptation often requires iteratively improving the model toward a targeted task, yet collecting …
Synthetic Data GenerationReinforcement Learning Based Sparse Black-box Adversarial Attack on Video Recognition Models
We explore the black-box adversarial attack on video recognition models. Attacks are only performed on selected key regions and key frames to reduce the high computation cost of searching adversarial perturbations on a v…
Adversarial Attackreinforcement-learningReinforcement Learning (RL)Saliency Detection+1