Unit Tests for Stochastic Optimization
Optimization by stochastic gradient descent is an important component of many large-scale machine learning algorithms. A wide variety of such optimization algorithms have been devised; however, it is unclear whether these algorithms are robust and widely applicable across many different optimization landscapes. In this paper we develop a collection of unit tests for stochastic optimization. Each unit test rapidly evaluates an optimization algorithm on a small-scale, isolated, and well-understood difficulty, rather than in real-world scenarios where many such issues are entangled. Passing these unit tests is not sufficient, but absolutely necessary for any algorithms with claims to generality or robustness. We give initial quantitative and qualitative results on numerous established algorithms. The testing framework is open-source, extensible, and easy to apply to new algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
Stochastic OptimizationSimilar Papers 제목 키워드 기반
On the Overlooked Structure of Stochastic Gradients
Stochastic gradients closely relate to both optimization and generalization of deep neural networks (DNNs). Some works attempted to explain the success of stochastic optimization for deep learning by the arguably heavy-t…
Deep LearningStochastic OptimizationSpanning Tests for Markowitz Stochastic Dominance
We derive properties of the cdf of random variables defined as saddle-type points of real valued continuous stochastic processes. This facilitates the derivation of the first-order asymptotic properties of tests for stoc…
ManagementInfluence of Stochastic Dependence on Network Constraints Screening for Unit Commitment
The deepening penetration of renewable energy is challenging how power system operators cope with the associated variability and uncertainty in the unit commitment problem. Given its computational complexity, several opt…
Block-Cyclic Stochastic Coordinate Descent for Deep Neural Networks
We present a stochastic first-order optimization algorithm, named BCSC, that adds a cyclic constraint to stochastic block-coordinate descent. It uses different subsets of the data to update different subsets of the param…
On the relationship between prediction intervals, tests of sharp nulls and inference on realized treatment effects in settings with few treated units
We study how inference methods for settings with few treated units that rely on treatment effect homogeneity extend to alternative inferential targets when treatment effects are heterogeneous -- namely, tests of sharp nu…
Prediction Intervalsvalid