Out-of-sample scoring and automatic selection of causal estimators
Recently, many causal estimators for Conditional Average Treatment Effect (CATE) and instrumental variable (IV) problems have been published and open sourced, allowing to estimate granular impact of both randomized treatments (such as A/B tests) and of user choices on the outcomes of interest. However, the practical application of such models has ben hampered by the lack of a valid way to score the performance of such models out of sample, in order to select the best one for a given application. We address that gap by proposing novel scoring approaches for both the CATE case and an important subset of instrumental variable problems, namely those where the instrumental variable is customer acces to a product feature, and the treatment is the customer's choice to use that feature. Being able to score model performance out of sample allows us to apply hyperparameter optimization methods to causal model selection and tuning. We implement that in an open source package that relies on DoWhy and EconML libraries for implementation of causal inference models (and also includes a Transformed Outcome model implementation), and on FLAML for hyperparameter optimization and for component models used in the causal models. We demonstrate on synthetic data that optimizing the proposed scores is a reliable method for choosing the model and its hyperparameter values, whose estimates are close to the true impact, in the randomized CATE and IV cases. Further, we provide examles of applying these methods to real customer data from Wise.
Code (1)
Tasks
Causal InferenceHyperparameter OptimizationModel SelectionSimilar Papers 제목 키워드 기반
Sample Selection Bias in Evaluation of Prediction Performance of Causal Models
Causal models are notoriously difficult to validate because they make untestable assumptions regarding confounding. New scientific experiments offer the possibility of evaluating causal models using prediction performanc…
PredictionSelection biasFeature selection in stratification estimators of causal effects: lessons from potential outcomes, causal diagrams, and structural equations
What is the ideal regression (if any) for estimating average causal effects? We study this question in the setting of discrete covariates, deriving expressions for the finite-sample variance of various stratification est…
feature selectionregressionOn Causal Inference with Model-Based Outcomes
We study a causal inference problem with group-level outcomes, which are themselves parameters identified from microdata. We formalize these outcomes using population moment conditions and demonstrate that one-step Gener…
Causal InferencemodelregressionOptimal Kernel Choice for Score Function-based Causal Discovery
Score-based methods have demonstrated their effectiveness in discovering causal relationships by scoring different causal structures based on their goodness of fit to the data. Recently, Huang et al. proposed a generaliz…
Causal DiscoveryLocal Constraint-Based Causal Discovery under Selection Bias
We consider the problem of discovering causal relations from independence constraints selection bias in addition to confounding is present. While the seminal FCI algorithm is sound and complete in this setup, no criterio…
Causal Discoveryscoring ruleSelection bias