AutoLoss: Learning Discrete Schedules for Alternate Optimization
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters. Appropriately scheduling the optimization of a task objective or a set of parameters is usually crucial to the quality of convergence. In this paper, we present AutoLoss, a meta-learning framework that automatically learns and determines the optimization schedule. AutoLoss provides a generic way to represent and learn the discrete optimization schedule from metadata, allows for a dynamic and data-driven schedule in ML problems that involve alternating updates of different parameters or from different loss objectives. We apply AutoLoss on four ML tasks: d-ary quadratic regression, classification using a multi-layer perceptron (MLP), image generation using GANs, and multi-task neural machine translation (NMT). We show that the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on all four tasks. The trained AutoLoss controller is generalizable -- it can guide and improve the learning of a new task model with different specifications, or on different datasets.
Code (1)
Tasks
Image GenerationMachine TranslationMeta-LearningNMTSchedulingTranslationSimilar Papers 제목 키워드 기반
AutoLoss: Learning Discrete Schedule for Alternate Optimization
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters. Appropriately scheduling the optimization of a task objective or a set…
Image GenerationMachine TranslationMeta-LearningNMT+1AutoLoss: Automated Loss Function Search in Recommendations
Designing an effective loss function plays a crucial role in training deep recommender systems. Most existing works often leverage a predefined and fixed loss function that could lead to suboptimal recommendation quality…
Recommendation SystemsAutoLossGen: Automatic Loss Function Generation for Recommender Systems
In recommendation systems, the choice of loss function is critical since a good loss may significantly improve the model performance. However, manually designing a good loss is a big challenge due to the complexity of th…
Recommendation SystemsLoss Function Discovery for Object Detection via Convergence-Simulation Driven Search
Designing proper loss functions for vision tasks has been a long-standing research direction to advance the capability of existing models. For object detection, the well-established classification and regression loss fun…
image-classificationModel Optimizationobject-detectionObject DetectionOpen-Ended Task Discovery via Bayesian Optimization
When applying Bayesian optimization (BO) to scientific workflow, a major yet often overlooked source of uncertainty is the task itself -- namely, what to optimize and how to evaluate it -- which can evolve as evidence ac…