Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study
The poor cross-architecture generalization of dataset distillation greatly weakens its practical significance. This paper attempts to mitigate this issue through an empirical study, which suggests that the synthetic datasets undergo an inductive bias towards the distillation model. Therefore, the evaluation model is strictly confined to having similar architectures of the distillation model. We propose a novel method of EvaLuation with distillation Feature (ELF), which utilizes features from intermediate layers of the distillation model for the cross-architecture evaluation. In this manner, the evaluation model learns from bias-free knowledge therefore its architecture becomes unfettered while retaining performance. By performing extensive experiments, we successfully prove that ELF can well enhance the cross-architecture generalization of current DD methods. Code of this project is at \url{https://github.com/Lirui-Zhao/ELF}.
Code (1)
Tasks
Dataset DistillationInductive BiasSimilar Papers 제목 키워드 기반
MetaDD: Boosting Dataset Distillation with Neural Network Architecture-Invariant Generalization
Dataset distillation (DD) entails creating a refined, compact distilled dataset from a large-scale dataset to facilitate efficient training. A significant challenge in DD is the dependency between the distilled dataset a…
Dataset DistillationPractical Knowledge Distillation: Using DNNs to Beat DNNs
For tabular data sets, we explore data and model distillation, as well as data denoising. These techniques improve both gradient-boosting models and a specialized DNN architecture. While gradient boosting is known to out…
DenoisingKnowledge DistillationMultimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis
Recent advances in multimodal learning have achieved remarkable success across diverse vision-language tasks. However, such progress heavily relies on large-scale image-text datasets, making training costly and inefficie…
D^4: Dataset Distillation via Disentangled Diffusion Model
Dataset distillation offers a lightweight synthetic dataset for fast network training with promising test accuracy. To imitate the performance of the original dataset most approaches employ bi-level optimization and …
Dataset DistillationD$^4$M: Dataset Distillation via Disentangled Diffusion Model
Dataset distillation offers a lightweight synthetic dataset for fast network training with promising test accuracy. To imitate the performance of the original dataset, most approaches employ bi-level optimization and the…
Dataset Distillation