GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost

2024-05-23 · Xinyi Shang, Peng Sun, Tao Lin

Recent advancements in dataset distillation have demonstrated the significant benefits of employing soft labels generated by pre-trained teacher models. In this paper, we introduce a novel perspective by emphasizing the full utilization of labels. We first conduct a comprehensive comparison of various loss functions for soft label utilization in dataset distillation, revealing that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization. This finding highlights the necessity of a universal loss function for training models on synthetic datasets. Building on these insights, we introduce an extremely simple yet surprisingly effective plug-and-play approach, GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to efficiently leverage full label information. Extensive experiments demonstrate that GIFT consistently enhances the state-of-the-art dataset distillation methods across various scales datasets without incurring additional computational costs. For instance, on ImageNet-1K with IPC = 10, GIFT improves the SOTA method RDED by 3.9% and 1.8% on ConvNet and ResNet-18, respectively. Code: https://github.com/LINs-lab/GIFT.

📄 PDF Abstract BibTeX arXiv:2405.14736

Code (1)

lins-lab/gift 공식 구현 pytorch

Tasks

Dataset Distillation

Rectified Decision Trees: Exploring the Landscape of Interpretable and Effective Machine Learning

2020-08-21 · Yiming Li, Jiawang Bai, Jiawei Li, Xue Yang 외

Interpretability and effectiveness are two essential and indispensable requirements for adopting machine learning methods in reality. In this paper, we propose a knowledge distillation based decision trees extension, dub…

BIG-bench Machine Learning Knowledge Distillation

A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning

2017-07-01 · CVPR 2017 7 · Junho Yim, Donggyu Joo, Jihoon Bae, Junmo Kim

We introduce a novel technique for knowledge transfer, where knowledge from a pretrained deep neural network (DNN) is distilled and transferred to another DNN. As the DNN performs a mapping from the input space to the ou…

Knowledge Distillation Transfer Learning

GiFT: Gibbs Fine-Tuning for Code Generation

2025-02-17 · Haochen Li, Wanjin Feng, Xin Zhou, Zhiqi Shen

Training Large Language Models (LLMs) with synthetic data is a prevalent practice in code generation. A key approach is self-training, where LLMs are iteratively trained on self-generated correct code snippets. In this c…

Code Generation valid

GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Models

2025-09-25 · Guowei Xu, Wenxin Xu, Jiawang Zhao, Kaisheng Ma arxiv

Diffusion models have recently shown strong potential in language modeling, offering faster generation compared to traditional autoregressive approaches. However, applying supervised fine-tuning (SFT) to diffusion models…

GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs

2024-08-27 · Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Egor Venediktov 외

Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance…

parameter-efficient fine-tuning Quantization