Determinantal Point Processes for Mini-Batch Diversification
We study a mini-batch diversification scheme for stochastic gradient descent (SGD). While classical SGD relies on uniformly sampling data points to form a mini-batch, we propose a non-uniform sampling scheme based on the Determinantal Point Process (DPP). The DPP relies on a similarity measure between data points and gives low probabilities to mini-batches which contain redundant data, and higher probabilities to mini-batches with more diverse data. This simultaneously balances the data and leads to stochastic gradients with lower variance. We term this approach Diversified Mini-Batch SGD (DM-SGD). We show that regular SGD and a biased version of stratified sampling emerge as special cases. Furthermore, DM-SGD generalizes stratified sampling to cases where no discrete features exist to bin the data into groups. We show experimentally that our method results more interpretable and diverse features in unsupervised setups, and in better classification accuracies in supervised setups.
Code (0)
등록된 구현이 없습니다.
Tasks
Point ProcessesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Active Mini-Batch Sampling using Repulsive Point Processes
The convergence speed of stochastic gradient descent (SGD) can be improved by actively selecting mini-batches. We explore sampling schemes where similar data points are less likely to be selected in the same mini-batch. …
Point ProcessesDeterminantal point process sampling for bioacoustic active learning
Eco-acoustic monitoring generates vast volumes of audio data, making active learning a promising approach for reducing annotation effort while efficiently training reliable biodiversity classifiers. This report presents …
Active LearningDeterminantal point processes based on orthogonal polynomials for sampling minibatches in SGD
Stochastic gradient descent (SGD) is a cornerstone of machine learning. When the number N of data items is large, SGD relies on constructing an unbiased estimator of the gradient of the empirical risk using a small subse…
Point ProcessesDiversified Sampling for Batched Bayesian Optimization with Determinantal Point Processes
In Bayesian Optimization (BO) we study black-box function optimization with noisy point evaluations and Bayesian priors. Convergence of BO can be greatly sped up by batching, where multiple evaluations of the black-box f…
Bayesian OptimizationDiversityPoint ProcessesThompson SamplingBatch Active Learning Using Determinantal Point Processes
Data collection and labeling is one of the main challenges in employing machine learning algorithms in a variety of real-world applications with limited data. While active learning methods attempt to tackle this issue by…
Active LearningDiversityPoint Processes