Learning Variational Models with Unrolling and Bilevel Optimization
In this paper we consider the problem of learning variational models in the context of supervised learning via risk minimization. Our goal is to provide a deeper understanding of the two approaches of learning of variational models via bilevel optimization and via algorithm unrolling. The former considers the variational model as a lower level optimization problem below the risk minimization problem, while the latter replaces the lower level optimization problem by an algorithm that solves said problem approximately. Both approaches are used in practice, but unrolling is much simpler from a computational point of view. To analyze and compare the two approaches, we consider a simple toy model, and compute all risks and the respective estimators explicitly. We show that unrolling can be better than the bilevel optimization approach, but also that the performance of unrolling can depend significantly on further parameters, sometimes in unexpected ways: While the stepsize of the unrolled algorithm matters a lot (and learning the stepsize gives a significant improvement), the number of unrolled iterations plays a minor role.
Code (0)
등록된 구현이 없습니다.
Tasks
Bilevel OptimizationRolling Shutter CorrectionSimilar Papers 제목 키워드 기반
An Optimization-Based Meta-Learning Model for MRI Reconstruction with Diverse Dataset
Purpose: This work aims at developing a generalizable MRI reconstruction model in the meta-learning framework. The standard benchmarks in meta-learning are challenged by learning on diverse task distributions. The propos…
Bilevel OptimizationMeta-LearningMRI ReconstructionRolling Shutter CorrectionUnderstanding the Curse of Unrolling
Algorithm unrolling is ubiquitous in machine learning, particularly in hyperparameter optimization and meta-learning, where Jacobians of solution mappings are computed by differentiating through iterative algorithms. Alt…
Hyperparameter OptimizationBilevel OptimizationEfficient Bilevel Optimization with KFAC-Based Hypergradients
Bilevel optimization (BO) is widely applicable to many machine learning problems. Scaling BO, however, requires repeatedly computing hypergradients, which involves solving inverse Hessian-vector products (IHVPs). In prac…
Bilevel OptimizationDelta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
Hyperparameter optimization of neural networks can be elegantly formulated as a bilevel optimization problem. While research on bilevel optimization of neural networks has been dominated by implicit differentiation and u…
Bilevel OptimizationHyperparameter OptimizationImage ClassificationRolling Shutter CorrectionTowards Understanding Graph Neural Networks: An Algorithm Unrolling Perspective
The graph neural network (GNN) has demonstrated its superior performance in various applications. The working mechanism behind it, however, remains mysterious. GNN models are designed to learn effective representations f…
Bilevel OptimizationDenoisingGraph Neural NetworkRolling Shutter Correction