GOOD: A Graph Out-of-Distribution Benchmark
Out-of-distribution (OOD) learning deals with scenarios in which training and test data follow different distributions. Although general OOD problems have been intensively studied in machine learning, graph OOD is only an emerging area of research. Currently, there lacks a systematic benchmark tailored to graph OOD method evaluation. In this work, we aim at developing an OOD benchmark, known as GOOD, for graphs specifically. We explicitly make distinctions between covariate and concept shifts and design data splits that accurately reflect different shifts. We consider both graph and node prediction tasks as there are key differences in designing shifts. Overall, GOOD contains 11 datasets with 17 domain selections. When combined with covariate, concept, and no shifts, we obtain 51 different splits. We provide performance results on 10 commonly used baseline methods with 10 random runs. This results in 510 dataset-model combinations in total. Our results show significant performance gaps between in-distribution and OOD settings. Our results also shed light on different performance trends between covariate and concept shifts by different methods. Our GOOD benchmark is a growing project and expects to expand in both quantity and variety of resources as the area develops. The GOOD benchmark can be accessed via https://github.com/divelab/GOOD/.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GOODAT: Towards Test-time Graph Out-of-Distribution Detection
Graph neural networks (GNNs) have found widespread application in modeling graph data across diverse domains. While GNNs excel in scenarios where the testing data shares the distribution of their training counterparts (i…
Out-of-Distribution DetectionDrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations
AI-aided drug discovery (AIDD) is gaining increasing popularity due to its promise of making the search for new pharmaceuticals quicker, cheaper and more efficient. In spite of its extensive use in many fields, such as A…
BenchmarkingDrug DiscoveryGraph Neural NetworkProtein FoldingGOOD-D: On Unsupervised Graph Out-Of-Distribution Detection
Most existing deep learning models are trained based on the closed-world assumption, where the test data is assumed to be drawn i.i.d. from the same distribution as the training data, known as in-distribution (ID). Howev…
Contrastive LearningData AugmentationOut-of-Distribution DetectionOut of Distribution (OOD) DetectionSGOOD: Substructure-enhanced Graph-Level Out-of-Distribution Detection
Graph-level representation learning is important in a wide range of applications. Existing graph-level models are generally built on i.i.d. assumption for both training and testing graphs. However, in an open world, mode…
Out-of-Distribution DetectionRepresentation LearningOut-of-Distribution Detection on Graphs: A Survey
Graph machine learning has witnessed rapid growth, driving advancements across diverse domains. However, the in-distribution assumption, where training and testing data share the same distribution, often breaks in real-w…
Anomaly DetectionGraph Anomaly DetectionOutlier DetectionOut-of-Distribution Detection+1