Memorization in Self-Supervised Learning Improves Downstream Generalization

2024-01-19 · Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, Franziska Boenisch

Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data-often scraped from the internet. This data can still be sensitive and empirical evidence suggests that SSL encoders memorize private information of their training data and can disclose them at inference time. Since existing theoretical definitions of memorization from supervised learning rely on labels, they do not transfer to SSL. To address this gap, we propose SSLMem, a framework for defining memorization within SSL. Our definition compares the difference in alignment of representations for data points and their augmented views returned by both encoders that were trained on these data points and encoders that were not. Through comprehensive empirical analysis on diverse encoder architectures and datasets we highlight that even though SSL relies on large datasets and strong augmentations-both known in supervised learning as regularization techniques that reduce overfitting-still significant fractions of training data points experience high memorization. Through our empirical results, we show that this memorization is essential for encoders to achieve higher generalization performance on different downstream tasks.

📄 PDF Abstract BibTeX arXiv:2401.12233

Code (1)

sprintml/memorizationssl 공식 구현 pytorch

Tasks

Memorization Self-Supervised Learning

On the Generalization and Causal Explanation in Self-Supervised Learning

2024-10-01 · Wenwen Qiang, Zeen Song, Ziyin Gu, Jiangmeng Li 외

Self-supervised learning (SSL) methods learn from unlabeled data and achieve high generalization performance on downstream tasks. However, they may also suffer from overfitting to their training data and lose the ability…

Memorization Self-Supervised Learning

Captured by Captions: On Memorization and its Mitigation in CLIP Models

2025-02-11 · Wenhao Wang, Adam Dziedzic, Grace C. Kim, Michael Backes 외

Multi-modal models, such as CLIP, have demonstrated strong performance in aligning visual and textual representations, excelling in tasks like image retrieval and zero-shot classification. Despite this success, the mecha…

Image Retrieval Memorization Self-Supervised Learning zero-shot-classification+1

Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization

2025-10-15 · Changsheng Wang, Xin Chen, Sijia Liu, Ke Ding arxiv

Adapting pretrained large language models (LLMs) to code domains via supervised fine-tuning (FT) has been commonly used for code generation. However, we identify a previously underappreciated failure mode, the memorizati…

Code Generation

Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles

2026-06-08 · Xiao Li, Yixuan Jia, Zekai Zhang, Xiang Li 외 arxiv

Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners, yet the connection between these two abilities remains less explored. Drawi…

Self-Supervised Learning

Unsupervised Meta-Learning via In-Context Learning

2024-05-25 · Anna Vettoruzzo, Lorenzo Braccaioli, Joaquin Vanschoren, Marlena Nowaczyk

Unsupervised meta-learning aims to learn feature representations from unsupervised datasets that can transfer to downstream tasks with limited labeled data. In this paper, we propose a novel approach to unsupervised meta…

In-Context Learning Memorization Meta-Learning