Representation Bayesian Risk Decompositions and Multi-Source Domain Adaptation
We consider representation learning (hypothesis class $\mathcal{H} = \mathcal{F}\circ\mathcal{G}$) where training and test distributions can be different. Recent studies provide hints and failure examples for domain invariant representation learning, a common approach for this problem, but the explanations provided are somewhat different and do not provide a unified picture. In this paper, we provide new decompositions of risk which give finer-grained explanations and clarify potential generalization issues. For Single-Source Domain Adaptation, we give an exact decomposition (an equality) of the target risk, via a natural hybrid argument, as sum of three factors: (1) source risk, (2) representation conditional label divergence, and (3) representation covariate shift. We derive a similar decomposition for the Multi-Source case. These decompositions reveal factors (2) and (3) as the precise reasons for failure to generalize. For example, we demonstrate that domain adversarial neural networks (DANN) attempt to regularize for (3) but miss (2), while a recent technique Invariant Risk Minimization (IRM) attempts to account for (2) but does not consider (3). We also verify our observations experimentally.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationRepresentation LearningSimilar Papers 제목 키워드 기반
Bayesian Spillover Graphs for Dynamic Networks
We present Bayesian Spillover Graphs (BSG), a novel method for learning temporal relationships, identifying critical nodes, and quantifying uncertainty for multi-horizon spillover effects in a dynamic system. BSG leverag…
Time SeriesTime Series AnalysisUncertainty QuantificationProfit and loss decomposition in continuous time and approximations
Financial institutions and insurance companies that analyze the evolution and sources of profits and losses often look at risk factors only at discrete reporting dates, ignoring the detailed paths. Continuous-time decomp…
On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes
Optimizing static risk-averse objectives in Markov decision processes is difficult because they do not admit standard dynamic programming equations common in Reinforcement Learning (RL) algorithms. Dynamic programming de…
Reinforcement Learning (RL)Predictive Uncertainty Quantification via Risk Decompositions for Strictly Proper Scoring Rules
Uncertainty quantification in predictive modeling often relies on ad hoc methods as there is no universally accepted formal framework for that. This paper introduces a theoretical approach to understanding uncertainty th…
Uncertainty QuantificationAre Random Decompositions all we need in High Dimensional Bayesian Optimisation?
Learning decompositions of expensive-to-evaluate black-box functions promises to scale Bayesian optimisation (BO) to high-dimensional problems. However, the success of these techniques depends on finding proper decomposi…
AllBayesian Optimisation