Distributionally Robust Optimization and Generalization in Kernel Methods
Distributionally robust optimization (DRO) has attracted attention in machine learning due to its connections to regularization, generalization, and robustness. Existing work has considered uncertainty sets based on phi-divergences and Wasserstein distances, each of which have drawbacks. In this paper, we study DRO with uncertainty sets measured via maximum mean discrepancy (MMD). We show that MMD DRO is roughly equivalent to regularization by the Hilbert norm and, as a byproduct, reveal deep connections to classic results in statistical learning. In particular, we obtain an alternative proof of a generalization bound for Gaussian kernel ridge regression via a DRO lense. The proof also suggests a new regularizer. Our results apply beyond kernel methods: we derive a generically applicable approximation of MMD DRO, and show that it generalizes recent work on variance-based regularization.
Code (1)
Similar Papers 제목 키워드 기반
A Distributionally Robust Optimization Method for Adversarial Multiple Kernel Learning
We propose a novel data-driven method to learn a mixture of multiple kernels with random features that is certifiabaly robust against adverserial inputs. Specifically, we consider a distributionally robust optimization o…
Generalization BoundsModel SelectionSemantic SegmentationSmall Data Image ClassificationKernel Distributionally Robust Optimization
We propose kernel distributionally robust optimization (Kernel DRO) using insights from the robust optimization theory and functional analysis. Our method uses reproducing kernel Hilbert spaces (RKHS) to construct a wide…
Stochastic OptimizationDistributionally Robust Optimal and Safe Control of Stochastic Systems via Kernel Conditional Mean Embedding
We present a novel distributionally robust framework for dynamic programming that uses kernel methods to design feedback control policies. Specifically, we leverage kernel mean embedding to map the transition probabiliti…
Distributional Robustness Bounds Generalization Errors
Bayesian methods, distributionally robust optimization methods, and regularization methods are three pillars of trustworthy machine learning combating distributional uncertainty, e.g., the uncertainty of an empirical dis…
Hedging Complexity in Generalization via a Parametric Distributionally Robust Optimization Framework
Empirical risk minimization (ERM) and distributionally robust optimization (DRO) are popular approaches for solving stochastic optimization problems that appear in operations management and machine learning. Existing gen…
Generalization BoundsManagementPortfolio OptimizationStochastic Optimization