Diametrical Risk Minimization: Theory and Computations
The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in sharp empirical risk landscapes from benchmark neural network classification problems with corrupted labels.
Code (1)
Tasks
Generalization BoundsSimilar Papers 제목 키워드 기반
Continual and Sliding Window Release for Private Empirical Risk Minimization
It is difficult to continually update private machine learning models with new data while maintaining privacy. Data incur increasing privacy loss -- as measured by differential privacy -- when they are used in repeated c…
Risk Minimization, Regret Minimization and Progressive Hedging Algorithms
This paper begins with a study on the dual representations of risk and regret measures and their impact on modeling multistage decision making under uncertainty. A relationship between risk envelopes and regret envelopes…
Decision MakingDecision Making Under UncertaintyInvariant Risk Minimization
We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions. To achieve this goal, IRM learns a data representation such that the optimal …
Domain GeneralizationImage ClassificationOut-of-Distribution GeneralizationSequential Counterfactual Risk Minimization
Counterfactual Risk Minimization (CRM) is a framework for dealing with the logged bandit feedback problem, where the goal is to improve a logging policy using offline data. In this paper, we explore the case where it is …
counterfactualDecision-making under risk: when is utility maximization equivalent to risk minimization?
Motivated by the analysis of a general optimal portfolio selection problem, which encompasses as special cases an optimal consumption and an optimal debt-arrangement problem, we are concerned with the questions of how a …
Decision Making