Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions
Graph generative models are a highly active branch of machine learning. Given the steady development of new models of ever-increasing complexity, it is necessary to provide a principled way to evaluate and compare them. In this paper, we enumerate the desirable criteria for such a comparison metric and provide an overview of the status quo of graph generative model comparison in use today, which predominantly relies on the maximum mean discrepancy (MMD). We perform a systematic evaluation of MMD in the context of graph generative model comparison, highlighting some of the challenges and pitfalls researchers inadvertently may encounter. After conducting a thorough analysis of the behaviour of MMD on synthetically-generated perturbed graphs as well as on recently-proposed graph generative models, we are able to provide a suitable procedure to mitigate these challenges and pitfalls. We aggregate our findings into a list of practical recommendations for researchers to use when evaluating graph generative models.
Code (2)
Similar Papers 제목 키워드 기반
Five Pitfalls When Assessing Synthetic Medical Images with Reference Metrics
Reference metrics have been developed to objectively and quantitatively compare two images. Especially for evaluating the quality of reconstructed or compressed images, these metrics have shown very useful. Extensive tes…
SSIMAre Heterophily-Specific GNNs and Homophily Metrics Really Effective? Evaluation Pitfalls and New Benchmarks
Over the past decade, Graph Neural Networks (GNNs) have achieved great success on machine learning tasks with relational data. However, recent studies have found that heterophily can cause significant performance degrada…
Demystifying Graph Neural Network Explanations
Graph neural networks (GNNs) are quickly becoming the standard approach for learning on graph structured data across several domains, but they lack transparency in their decision-making. Several perturbation-based approa…
Decision MakingGraph Neural NetworkSynthetic Data GenerationA Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges
Graph Neural Networks (GNNs) perform well in community detection and molecule classification. Counterfactual Explanations (CE) provide counter-examples to overcome the transparency limitations of black-box models. Due to…
BenchmarkingCommunity DetectioncounterfactualCounterfactual Explanation+3The Jungle of Generative Drug Discovery: Traps, Treasures, and Ways Out
"How to evaluate de novo designs proposed by a generative model?" Despite the transformative potential of generative deep learning in drug discovery, this seemingly simple question has no clear answer. The absence of sta…
BenchmarkingDeep LearningDrug DiscoveryNavigate