Correcting Flaws in Common Disentanglement Metrics
Recent years have seen growing interest in learning disentangled representations, in which distinct features, such as size or shape, are represented by distinct neurons. Quantifying the extent to which a given representation is disentangled is not straightforward; multiple metrics have been proposed. In this paper, we identify two failings of existing metrics, which mean they can assign a high score to a model which is still entangled, and we propose two new metrics, which redress these problems. We then consider the task of compositional generalization. Unlike prior works, we treat this as a classification problem, which allows us to use it to measure the disentanglement ability of the encoder, without depending on the decoder. We show that performance on this task is (a) generally quite poor, (b) correlated with most disentanglement metrics, and (c) most strongly correlated with our newly proposed metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderDisentanglementSimilar Papers 제목 키워드 기반
Identifying and Mitigating Flaws of Deep Perceptual Similarity Metrics
Measuring the similarity of images is a fundamental problem to computer vision for which no universal solution exists. While simple metrics such as the pixel-wise L2-norm have been shown to have significant flaws, they r…
Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods
Multiple-choice questions with item-writing flaws can negatively impact student learning and skew analytics. These flaws are often present in student-generated questions, making it difficult to assess their quality and s…
Multiple-choiceReward (Mis)design for Autonomous Driving
This article considers the problem of diagnosing certain common errors in reward design. Its insights are also applicable to the design of cost functions and performance metrics more generally. To diagnose common errors,…
Autonomous Drivingreinforcement-learningReinforcement Learning (RL)An Automatic Question Usability Evaluation Toolkit
Evaluating multiple-choice questions (MCQs) involves either labor intensive human assessments or automated methods that prioritize readability, often overlooking deeper question design flaws. To address this issue, we in…
Multiple-choiceWord EmbeddingsMeasuring the Effect of Causal Disentanglement on the Adversarial Robustness of Neural Network Models
Causal Neural Network models have shown high levels of robustness to adversarial attacks as well as an increased capacity for generalisation tasks such as few-shot learning and rare-context classification compared to tra…
Adversarial RobustnessBenchmarkingDisentanglementFew-Shot Learning+2