Recovering the Lowest Layer of Deep Networks with High Threshold Activations
Giving provable guarantees for learning neural networks is a core challenge of machine learning theory. Most prior work gives parameter recovery guarantees for one hidden layer networks, however, the networks used in practice have multiple non-linear layers. In this work, we show how we can strengthen such results to deeper networks -- we address the problem of uncovering the lowest layer in a deep neural network under the assumption that the lowest layer uses a high threshold before applying the activation, the upper network can be modeled as a well-behaved polynomial and the input distribution is Gaussian.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningLearning TheoryVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
Globally Optimal Training of Neural Networks with Threshold Activation Functions
Threshold activation functions are highly preferable in neural networks due to their efficiency in hardware implementations. Moreover, their mode of operation is more interpretable and resembles that of biological neuron…
Memory capacity of neural networks with threshold and ReLU activations
Overwhelming theoretical and empirical evidence shows that mildly overparametrized neural networks -- those with more connections than the size of the training data -- are often able to memorize the training data with $1…
Open-Ended Question AnsweringZoom-CAM: Generating Fine-grained Pixel Annotations from Image Labels
Current weakly supervised object localization and segmentation rely on class-discriminative visualization techniques to generate pseudo-labels for pixel-level training. Such visualization methods, including class activat…
Object LocalizationSegmentationSemantic SegmentationWeakly-Supervised Object Localization+2Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models
Efficient inference in Large Language Models (LLMs) requires deciding where computation can be reduced while preserving model quality. We study this problem through multilayer perceptron (MLP) activation sparsification a…
Considering Layerwise Importance in the Lottery Ticket Hypothesis
The Lottery Ticket Hypothesis (LTH) showed that by iteratively training a model, removing connections with the lowest global weight magnitude and rewinding the remaining connections, sparse networks can be extracted. Thi…