Scalable Nested Optimization for Deep Learning
Gradient-based optimization has been critical to the success of machine learning, updating a single set of parameters to minimize a single loss. A growing number of applications rely on a generalization of this, where we have a bilevel or nested optimization of which subsets of parameters update on different objectives nested inside each other. We focus on motivating examples of hyperparameter optimization and generative adversarial networks. However, naively applying classical methods often fails when we look at solving these nested problems on a large scale. In this thesis, we build tools for nested optimization that scale to deep learning setups.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningHyperparameter OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Nested Aggregates in Answer Sets: An Application to a Priori Optimization
We allow representing and reasoning in the presence of nested multiple aggregates over multiple variables and nested multiple aggregates over functions involving multiple variables in answer sets, precisely, in answer se…
Traveling Salesman ProblemIntegration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models
The variability and biases in the real-world performance benchmarking of deep learning models for medical imaging compromise their trustworthiness for real-world deployment. The common approach of holding out a single fi…
BenchmarkingHyperparameter OptimizationA Neural Transition-based Model for Nested Mention Recognition
It is common that entity mentions can contain other mentions recursively. This paper introduces a scalable transition-based method to model the nested structure of mentions. We first map a sentence with nested mentions t…
Named Entity Recognition (NER)Nested Mention RecognitionNested Named Entity RecognitionSentenceConstructing unbiased gradient estimators with finite variance for conditional stochastic optimization
We study stochastic gradient descent for solving conditional stochastic optimization problems, in which an objective to be minimized is given by a parametric nested expectation with an outer expectation taken with respec…
Stochastic OptimizationTighter Analysis of Alternating Stochastic Gradient Method for Stochastic Nested Problems
Stochastic nested optimization, including stochastic compositional, min-max and bilevel optimization, is gaining popularity in many machine learning applications. While the three problems share the nested structure, exis…
Bilevel Optimization