Assessing model error in counterfactual worlds
Counterfactual scenario modeling exercises that ask "what would happen if?" are one of the most common ways we plan for the future. Despite their ubiquity in planning and decision making, scenario projections are rarely evaluated retrospectively. Differences between projections and observations come from two sources: scenario deviation and model miscalibration. We argue the latter is most important for assessing the value of models in decision making, but requires estimating model error in counterfactual worlds. Here we present and contrast three approaches for estimating this error, and demonstrate the benefits and limitations of each in a simulation experiment. We provide recommendations for the estimation of counterfactual error and discuss the components of scenario design that are required to make scenario projections evaluable.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingSimilar Papers 제목 키워드 기반
Counterfactuals Modulo Temporal Logics
Lewis' theory of counterfactuals is the foundation of many contemporary notions of causality. In this paper, we extend this theory in the temporal direction to enable symbolic counterfactual reasoning on infinite sequenc…
counterfactualCounterfactual ReasoningA Dynamic Semantics for Causal Counterfactuals
Under the standard approach to counterfactuals, to determine the meaning of a counterfactual sentence, we consider the {``}closest{''} possible world(s) where the antecedent is true, and evaluate the consequent. Building…
counterfactualSentenceCounterfactual Policy Evaluation for Decision-Making in Autonomous Driving
Learning-based approaches, such as reinforcement and imitation learning are gaining popularity in decision-making for autonomous driving. However, learned policies often fail to generalize and cannot handle novel situati…
Autonomous DrivingAutonomous VehiclescounterfactualDecision Making+2WorldKernel: A World Model is the Coupling Kernel of Admissible Possible Worlds
A common assumption holds that enough observational and interventional data, given to a strong enough predictor, suffices. We report a failure mode that contradicts it. Across hundreds of structural causal models, on ide…
Ceteris paribus logic in counterfactual reasoning
The semantics for counterfactuals due to David Lewis has been challenged on the basis of unlikely, or impossible, events. Such events may skew a given similarity order in favour of those possible worlds which exhibit the…
counterfactualCounterfactual ReasoningDecision Making