Laplace Sample Information: Data Informativeness Through a Bayesian Lens
Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset. We experimentally show that LSI is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models.
Code (1)
Tasks
InformativenessSimilar Papers 제목 키워드 기반
Measuring Sample Importance in Data Pruning for Language Models based on Information Entropy
Compute-efficient training of language models has become an important issue. We consider data pruning for data-efficient training of LLMs. In this work, we consider a data pruning method based on information entropy. We …
Data CompressionInformativenessLanguage ModelingLanguage ModellingDoes Informativeness Matter? Active Learning for Educational Dialogue Act Classification
Dialogue Acts (DAs) can be used to explain what expert tutors do and what students know during the tutoring process. Most empirical studies adopt the random sampling method to obtain sentence samples for manual annotatio…
Active LearningDialogue Act ClassificationInformativenessSentenceGrounding and Enhancing Informativeness and Utility in Dataset Distillation
Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship betwee…
Knowledge DistillationRiemannian Laplace approximations for Bayesian neural networks
Bayesian neural networks often approximate the weight-posterior with a Gaussian distribution. However, practical posteriors are often, even locally, highly non-Gaussian, and empirical performance deteriorates. We propose…
Identifying Memorization of Diffusion Models through p-Laplace Analysis
Diffusion models, today's leading image generative models, estimate the score function, i.e. the gradient of the log probability of (perturbed) data samples, without direct access to the underlying probability distributi…
Memorization