Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction

2021-10-28 · NeurIPS 2021 12 · Konstantin Schürholt, Dimche Kostadinov, Damian Borth

Self-Supervised Learning (SSL) has been shown to learn useful and information-preserving representations. Neural Networks (NNs) are widely applied, yet their weight space is still not fully understood. Therefore, we propose to use SSL to learn hyper-representations of the weights of populations of NNs. To that end, we introduce domain specific data augmentations and an adapted attention architecture. Our empirical evaluation demonstrates that self-supervised representation learning in this domain is able to recover diverse NN model characteristics. Further, we show that the proposed learned representations outperform prior work for predicting hyper-parameters, test accuracy, and generalization gap as well as transfer to out-of-distribution settings.

📄 PDF Abstract BibTeX arXiv:2110.15288

Code (1)

hsg-aiml/neurips_2021-weight_space_learning 공식 구현 pytorch

Tasks

Representation Learning Self-Supervised Learning

Methods 이 논문이 사용한 방법론

Test 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

InfoNCE 설명 없음

Contrastive Multiview Coding Contrastive Multiview Coding (CMC) is a self-supervised learning approach, based on CPC, that learns…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Hyper-Representations: Learning from Populations of Neural Networks

2024-10-07 · Konstantin Schürholt

This thesis addresses the challenge of understanding Neural Networks through the lens of their most fundamental component: the weights, which encapsulate the learned information and determine the model behavior. At the c…

Representation Learning Transfer Learning

Hyperspherically Regularized Networks for Self-Supervision

2021-04-29 · Aiden Durrant, Georgios Leontidis

Bootstrap Your Own Latent (BYOL) introduced an approach to self-supervised learning avoiding the contrastive paradigm and subsequently removing the computational burden of negative sampling associated with such methods. …

Diversity Self-Supervised Learning

Self-supervised Contrastive Learning for Cross-domain Hyperspectral Image Representation

2022-02-08 · Hyungtae Lee, Heesung Kwon

Recently, self-supervised learning has attracted attention due to its remarkable ability to acquire meaningful representations for classification tasks without using semantic labels. This paper introduces a self-supervis…

Contrastive Learning Self-Supervised Learning Transfer Learning

HyperNet: Self-Supervised Hyperspectral Spatial-Spectral Feature Understanding Network for Hyperspectral Change Detection

2022-07-20 · Meiqi Hu, Chen Wu, Liangpei Zhang

The fast development of self-supervised learning lowers the bar learning feature representation from massive unlabeled data and has triggered a series of research on change detection of remote sensing images. Challenges …

Change Detection Self-Supervised Learning

Self-Supervised Pretraining for Heterogeneous Hypergraph Neural Networks

2023-11-19 · Abdalgader Abubaker, Takanori Maehara, Madhav Nimishakavi, Vassilis Plachouras

Recently, pretraining methods for the Graph Neural Networks (GNNs) have been successful at learning effective representations from unlabeled graph data. However, most of these methods rely on pairwise relations in the gr…

Link Prediction Node Classification