CompRess: Self-Supervised Learning by Compressing Representations

2020-10-28 · NeurIPS 2020 12 · Soroush Abbasi Koohpayegani, Ajinkya Tejankar, Hamed Pirsiavash

Self-supervised learning aims to learn good representations with unlabeled data. Recent works have shown that larger models benefit more from self-supervised learning than smaller models. As a result, the gap between supervised and self-supervised learning has been greatly reduced for larger models. In this work, instead of designing a new pseudo task for self-supervised learning, we develop a model compression method to compress an already learned, deep self-supervised model (teacher) to a smaller one (student). We train the student model so that it mimics the relative similarity between the data points in the teacher's embedding space. For AlexNet, our method outperforms all previous methods including the fully supervised model on ImageNet linear evaluation (59.0% compared to 56.5%) and on nearest neighbor evaluation (50.7% compared to 41.4%). To the best of our knowledge, this is the first time a self-supervised AlexNet has outperformed supervised one on ImageNet classification. Our code is available here: https://github.com/UMBCvision/CompRess

📄 PDF Abstract BibTeX arXiv:2010.14713

Code (1)

UMBCvision/CompReSS 공식 구현 pytorch

Tasks

Linear evaluationModel Compression Self-Supervised Image Classification

Once-for-All Sequence Compression for Self-Supervised Speech Models

2022-11-04 · Hsuan-Jui Chen, Yen Meng, Hung-Yi Lee

The sequence length along the time axis is often the dominant factor of the computation in speech processing. Works have been proposed to reduce the sequence length for lowering the computational cost in self-supervised …

All

On Compressing Sequences for Self-Supervised Speech Models

2022-10-13 · Yen Meng, Hsuan-Jui Chen, Jiatong Shi, Shinji Watanabe 외

Compressing self-supervised models has become increasingly necessary, as self-supervised models become larger. While previous approaches have primarily focused on compressing the model size, shortening sequences is also …

Self-Supervised Learning

Supervised Compression for Resource-Constrained Edge Computing Systems

2021-08-21 · Yoshitomo Matsubara, Ruihan Yang, Marco Levorato, Stephan Mandt

There has been much interest in deploying deep learning algorithms on low-powered devices, including smartphones, drones, and medical sensors. However, full-scale deep neural networks are often too resource-intensive in …

Data Compression Edge-computing Feature Compression Image Compression+1

Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers

2022-11-17 · Tzu-Quan Lin, Tsung-Huan Yang, Chun-Yao Chang, Kuang-Ming Chen 외

Transformer-based self-supervised models have achieved remarkable success in speech processing, but their large size and high inference cost present significant challenges for real-world deployment. While numerous compre…

Knowledge Distillation Model Compression Self-Supervised Learning

MAPLE: Self-Supervised Learning-Enhanced Nonlinear Dimensionality Reduction for Visual Analysis

2026-01-28 · Zeyang Huang, Takanori Fujiwara, Angelos Chatzimparmpas, Wandrille Duchemin 외 arxiv

We present a new nonlinear dimensionality reduction method, MAPLE, that enhances UMAP by improving manifold modeling. MAPLE employs a self-supervised learning approach to more efficiently encode low-dimensional manifold …

Self-Supervised Learning Dimensionality Reduction