Submodular Batch Selection for Training Deep Neural Networks
Mini-batch gradient descent based methods are the de facto algorithms for training neural network architectures today. We introduce a mini-batch selection strategy based on submodular function maximization. Our novel submodular formulation captures the informativeness of each sample and diversity of the whole subset. We design an efficient, greedy algorithm which can give high-quality solutions to this NP-hard combinatorial optimization problem. Our extensive experiments on standard datasets show that the deep models trained using the proposed batch selection strategy provide better generalization than Stochastic Gradient Descent as well as a popular baseline sampling strategy across different learning rates, batch sizes, and distance metrics.
Code (1)
Tasks
Combinatorial OptimizationDiversityInformativenessSimilar Papers 제목 키워드 기반
Partial-Adaptive Submodular Maximization
The goal of a typical adaptive sequential decision making problem is to design an interactive policy that selects a group of items sequentially, based on some partial observations, to maximize the expected utility. It ha…
Active LearningDecision MakingSequential Decision MakingMinibatch Selection for Language Models via Partition Matroid Constrained Gradient Matching
Training large language models (LLMs) on heterogeneous data requires selecting minibatches that balance convergence speed with coverage across domains. Existing methods either select samples independently within each dom…
Active Learning Methods based on Statistical Leverage Scores
In many real-world machine learning applications, unlabeled data are abundant whereas class labels are expensive and scarce. An active learner aims to obtain a model of high accuracy with as few labeled instances as poss…
Active LearningOnline Summarization via Submodular and Convex Optimization
We consider the problem of subset selection in the online setting, where data arrive incrementally. Instead of storing and running subset selection on the entire dataset, we propose an incremental subset selection framew…
Video SummarizationSubmodular Mini-Batch Training in Generative Moment Matching Networks
This article was withdrawn because (1) it was uploaded without the co-authors' knowledge or consent, and (2) there are allegations of plagiarism.