On the Generalization and Causal Explanation in Self-Supervised Learning
Self-supervised learning (SSL) methods learn from unlabeled data and achieve high generalization performance on downstream tasks. However, they may also suffer from overfitting to their training data and lose the ability to adapt to new tasks. To investigate this phenomenon, we conduct experiments on various SSL methods and datasets and make two observations: (1) Overfitting occurs abruptly in later layers and epochs, while generalizing features are learned in early layers for all epochs; (2) Coding rate reduction can be used as an indicator to measure the degree of overfitting in SSL models. Based on these observations, we propose Undoing Memorization Mechanism (UMM), a plug-and-play method that mitigates overfitting of the pre-trained feature extractor by aligning the feature distributions of the early and the last layers to maximize the coding rate reduction of the last layer output. The learning process of UMM is a bi-level optimization process. We provide a causal analysis of UMM to explain how UMM can help the pre-trained feature extractor overcome overfitting and recover generalization. We also demonstrate that UMM significantly improves the generalization performance of SSL methods on various downstream tasks.
Code (1)
Tasks
MemorizationSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Invariant Structure Learning for Better Generalization and Causal Explainability
Learning the causal structure behind data is invaluable for improving generalization and obtaining high-quality explanations. We propose a novel framework, Invariant Structure Learning (ISL), that is designed to improve …
Self-Supervised LearningRepresentation Learning via Invariant Causal Mechanisms
Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signal by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks w…
Contrastive LearningOut-of-Distribution GeneralizationRepresentation LearningSelf-Supervised Image Classification+1On the Out-of-Distribution Generalization of Self-Supervised Learning
In this paper, we focus on the out-of-distribution (OOD) generalization of self-supervised learning (SSL). By analyzing the mini-batch construction during the SSL training phase, we first give one plausible explanation f…
Causal InferenceOut-of-Distribution GeneralizationSelf-Supervised LearningQuantifying Explanation Quality in Graph Neural Networks using Out-of-Distribution Generalization
Evaluating the quality of post-hoc explanations for Graph Neural Networks (GNNs) remains a significant challenge. While recent years have seen an increasing development of explainability methods, current evaluation metri…
Test-Time Learning of Causal Structure from Interventional Data
Supervised causal learning has shown promise in causal discovery, yet it often struggles with generalization across diverse interventional settings, particularly when intervention targets are unknown. To address this, we…
Causal Inference