Learning distinct features helps, provably
We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average $L_2$-distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network's units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityGeneralization BoundsSimilar Papers 제목 키워드 기반
Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning
Self-supervised feature learning and pretraining methods in reinforcement learning (RL) often rely on information-theoretic principles, termed mutual information skill learning (MISL). These methods aim to learn a repres…
Representation LearningReinforcement LearningTowards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks
For unsupervised pretraining, mask-reconstruction pretraining (MRP) approaches, e.g. MAE and data2vec, randomly mask input patches and then reconstruct the pixels or semantic features of these masked patches via an auto-…
DecoderIs Supervised Learning With Adversarial Features Provably Better Than Sole Supervision?
Generative Adversarial Networks (GAN) have shown promising results on a wide variety of complex tasks. Recent experiments show adversarial training provides useful gradients to the generator that helps attain better perf…
Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning
Self-supervised learning enables networks to learn discriminative features from massive data itself. Most state-of-the-art methods maximize the similarity between two augmentations of one image based on contrastive learn…
ClusteringContrastive LearningRelationSelf-Supervised LearningD-Net: Learning for Distinctive Point Clouds by Self-Attentive Point Searching and Learnable Feature Fusion
Learning and selecting important points on a point cloud is crucial for point cloud understanding in various applications. Most of early methods selected the important points on 3D shapes by analyzing the intrinsic geome…