홈 › Papers

Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks

2024-07-17 · Antoni Kowalczuk, Jan Dubiński, Atiyeh Ashari Ghomi, Yi Sui, George Stein, Jiapeng Wu, Jesse C. Cresswell, Franziska Boenisch, Adam Dziedzic

Large-scale vision models have become integral in many applications due to their unprecedented performance and versatility across downstream tasks. However, the robustness of these foundation models has primarily been explored for a single task, namely image classification. The vulnerability of other common vision tasks, such as semantic segmentation and depth estimation, remains largely unknown. We present a comprehensive empirical evaluation of the adversarial robustness of self-supervised vision encoders across multiple downstream tasks. Our attacks operate in the encoder embedding space and at the downstream task output level. In both cases, current state-of-the-art adversarial fine-tuning techniques tested only for classification significantly degrade clean and robust performance on other tasks. Since the purpose of a foundation model is to cater to multiple applications at once, our findings reveal the need to enhance encoder robustness more broadly. Our code is available at ${github.com/layer6ai-labs/ssl-robustness}$.

📄 PDF Abstract BibTeX arXiv:2407.12588

Code (1)

layer6ai-labs/ssl-robustness 공식 구현 pytorch

Tasks

Adversarial Robustness Benchmarking Depth Estimation image-classification Image Classification Self-Supervised Learning Semantic Segmentation

RoFt-Mol: Benchmarking Robust Fine-Tuning with Molecular Graph Foundation Models

2025-08-30 · Shikun Liu, Deyu Zou, Nima Shoghi, Victor Fung 외 arxiv

In the era of foundation models, fine-tuning pre-trained models for specific downstream tasks has become crucial. This drives the need for robust fine-tuning methods to address challenges such as model overfitting and sp…

Benchmarking Self-Supervised Contrastive Learning Methods for Image-Based Plant Phenotyping

2023-03-01 · Plant Phenomics 2023 3 · Franklin Ogidi, Mark Eramian, Ian Stavness

The rise of self-supervised learning (SSL) methods in recent years presents an opportunity to leverage unlabeled and domain-specific datasets generated by image-based plant phenotyping platforms to accelerate plant breed…

Benchmarking Contrastive Learning Head Detection Plant Phenotyping+1

Benchmarking Self-Supervised Learning on Diverse Pathology Datasets

2022-12-09 · CVPR 2023 1 · Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo 외

Computational pathology can lead to saving human lives, but models are annotation hungry and pathology images are notoriously expensive to annotate. Self-supervised learning has shown to be an effective method for utiliz…

Benchmarking Classification Instance Segmentation Self-Supervised Learning+1

Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?

2023-06-01 · Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid 외

Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance on speech tasks using only small amounts of annotated data. The high number of pro…

Benchmarking Decoder Self-Supervised Learning

Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Heads

2023-08-28 · Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid 외

Self-supervised learning (SSL) leverages large datasets of unlabeled speech to reach impressive performance with reduced amounts of annotated data. The high number of proposed approaches fostered the emergence of compreh…

Benchmarking Self-Supervised Learning