Inferring Capabilities from Task Performance with Bayesian Triangulation
As machine learning models become more general, we need to characterise them in richer, more meaningful ways. We describe a method to infer the cognitive profile of a system from diverse experimental data. To do so, we introduce measurement layouts that model how task-instance features interact with system capabilities to affect performance. These features must be triangulated in complex ways to be able to infer capabilities from non-populational data -- a challenge for traditional psychometric and inferential tools. Using the Bayesian probabilistic programming library PyMC, we infer different cognitive profiles for agents in two scenarios: 68 actual contestants in the AnimalAI Olympics and 30 synthetic agents for O-PIAAGETS, an object permanence battery. We showcase the potential for capability-oriented evaluation.
Code (0)
등록된 구현이 없습니다.
Tasks
Probabilistic ProgrammingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Bayesian Networks, Markov Networks, Moralisation, Triangulation: a Categorical Perspective
Moralisation and Triangulation are transformations allowing to switch between different ways of factoring a probability distribution into a graphical model. Moralisation allows to view a Bayesian network (a directed mode…
An Algebraic Approach to Moralisation and Triangulation of Probabilistic Graphical Models
Moralisation and Triangulation are transformations allowing to switch between different ways of factoring a probability distribution into a graphical model. Moralisation allows to view a Bayesian network (a directed mode…
Proving the NP-completeness of optimal moral graph triangulation
Moral graphs were introduced in the 1980s as an intermediate step when transforming a Bayesian network to a junction tree, on which exact belief propagation can be efficiently done. The moral graph of a Bayesian network …
Triangulation candidates for Bayesian optimization
Bayesian optimization involves "inner optimization" over a new-data acquisition criterion which is non-convex/highly multi-modal, may be non-differentiable, or may otherwise thwart local numerical optimizers. In such cas…
Bayesian OptimizationThe DNA of Calabi-Yau Hypersurfaces
We implement Genetic Algorithms for triangulations of four-dimensional reflexive polytopes which induce Calabi-Yau threefold hypersurfaces via Batryev's construction. We demonstrate that such algorithms efficiently optim…
Bayesian Optimization