Learning with Shallow Neural Networks on Cluster-Structured Features
The success of deep learning in high-dimensional settings is often attributed to the presence of low-dimensional structure in real-world data. While standard theoretical models typically assume that this structure lies in the target function, projecting unstructured inputs onto a low-dimensional subspace, data such as images, text or genomic sequences exhibit strong spatial correlations within the input space itself. In this paper, we propose a tractable model to study how these correlations affect the sample complexity of learning with gradient descent on shallow neural networks. Specifically, we consider targets that depend on a small number of latent Boolean variables, and input features grouped into clusters and correlated with the latent variables. Under an identifiability assumption, we show that for a layerwise gradient-descent variant, the sample complexity scales with the number of hidden variables and, when the signal-to-noise ratio is sufficiently high, is independent of the input dimension, up to logarithmic terms. We empirically test our theoretical findings on both synthetic and real data.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Asymmetric double-winged multi-view clustering network for exploring Diverse and Consistent Information
In unsupervised scenarios, deep contrastive multi-view clustering (DCMVC) is becoming a hot research spot, which aims to mine the potential relationships between different views. Most existing DCMVC algorithms focus on e…
ClusteringDiversityPseudo LabelOn the convergence of group-sparse autoencoders
Recent approaches in the theoretical analysis of model-based deep learning architectures have studied the convergence of gradient descent in shallow ReLU networks that arise from generative models whose hidden layers are…
ClusteringRobust Multilingual Named Entity Recognition with Shallow Semi-Supervised Features
We present a multilingual Named Entity Recognition approach based on a robust and general set of features across languages and datasets. Our system combines shallow local information with clustering semi-supervised featu…
ClusteringMultilingual Named Entity Recognitionnamed-entity-recognitionNamed Entity Recognition+1The Rlign Algorithm for Enhanced Electrocardiogram Analysis through R-Peak Alignment for Explainable Classification and Clustering
Electrocardiogram (ECG) recordings have long been vital in diagnosing different cardiac conditions. Recently, research in the field of automatic ECG processing using machine learning methods has gained importance, mainly…
ClusteringQuantum Feature Optimization for Enhanced Clustering of Blockchain Transaction Data
Blockchain transaction data exhibits high dimensionality, noise, and intricate feature entanglement, presenting significant challenges for traditional clustering algorithms. In this study, we conduct a comparative analys…
Clustering