Exploring Learning Complexity for Efficient Downstream Dataset Pruning

2024-02-08 · Wenyu Jiang, Zhenlong Liu, Zejian Xie, Songxin Zhang, BingYi Jing, Hongxin Wei

The ever-increasing fine-tuning cost of large-scale pre-trained models gives rise to the importance of dataset pruning, which aims to reduce dataset size while maintaining task performance. However, existing dataset pruning methods require training on the entire dataset, which is impractical for large-scale pre-trained models. In this paper, we propose a straightforward, novel, and training-free hardness score named Distorting-based Learning Complexity (DLC), to identify informative images and instructions from the downstream dataset efficiently. Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters. Specifically, we define the Learning Complexity to quantify sample hardness and utilize a lightweight weights masking process for fast estimation, instead of the costly SGD optimization. Based on DLC, we further design a flexible under-sampling with randomness (dubbed FlexRand), replacing the top-K strategy, to alleviate the severe subset distribution shift. Extensive experiments with downstream image and instructions dataset pruning benchmarks demonstrate the effectiveness and efficiency of the proposed approach. In the images pruning benchmark, DLC significantly reduces the pruning time by 35x while establishing state-of-the-art performance with FlexRand.

📄 PDF Abstract BibTeX arXiv:2402.05356

Code (0)

등록된 구현이 없습니다.

Tasks

Informativeness

Methods 이 논문이 사용한 방법론

Dataset Pruning Dataset pruning is an approach to reduce a large dataset to obtain a small dataset by removing less significant sample.

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Pruning 설명 없음

Exploring 3D Dataset Pruning

2026-02-28 · Xiaohan Zhao, Xinyi Shang, Jiacheng Liu, Zhiqiang Shen arxiv

Dataset pruning has been widely studied for 2D images to remove redundancy and accelerate training, while particular pruning methods for 3D data remain largely unexplored. In this work, we study dataset pruning for 3D da…

Learning from Complexity: Exploring Dynamic Sample Pruning of Spatio-Temporal Training

2026-02-22 · Wei Chen, Junle Chen, Yuqian Wu, Yuxuan Liang 외 arxiv

Spatio-temporal forecasting is fundamental to intelligent systems in transportation, climate science, and urban planning. However, training deep learning models on the massive, often redundant, datasets from these domain…

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning

2024-10-09 · Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Kumar Jaiswal 외

Network pruning has emerged as a potential solution to make LLMs cheaper to deploy. However, existing LLM pruning approaches universally rely on the C4 dataset as the calibration data for calculating pruning scores, leav…

In-Context Learning Network Pruning

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models

2024-05-30 · Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion 외

In this work, we investigate whether small language models can determine high-quality subsets of large-scale text datasets that improve the performance of larger language models. While existing work has shown that prunin…

Accelerating Large Scale Real-Time GNN Inference using Channel Pruning

2021-05-10 · Hongkuan Zhou, Ajitesh Srivastava, Hanqing Zeng, Rajgopal Kannan 외

Graph Neural Networks (GNNs) are proven to be powerful models to generate node embedding for downstream applications. However, due to the high computation complexity of GNN inference, it is hard to deploy GNNs for large-…

CPU GPU Node Classification Spam detection