Tight Frame Contractions in Deep Networks
Numerical experiments demonstrate that deep neural networks classifiers progressively separate class distributions around their mean, achieving linear separability. To explain this mechanism, we introduce structured deep network architectures that can be analyzed mathematically and have high classification accuracies on complex image databases. They iterate over tight frame contractions, which apply a pointwise contraction on decomposition coefficients in a tight frame. Tight frame contractions can reduce within-class variabilities while preserving class mean separations and hence improve the Fisher discriminant ratio. Variance reduction bounds are proved for soft-thresholding contractions with Gaussian mixture models. Iterating on tight frame contractions defines deep convolutional network without bias parameters in hidden layers. We show that spatial filters do not need to be learned, and can be defined from wavelet frames. Learning frame contractions along the resulting wavelet scattering channels is sufficient to nearly reach the classification accuracies of VGG-11 and ResNet-18 on ImageNet, with no learned bias.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference
Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models. In the moderate-sparsity regime, Gustavson's da…
ABB-BERT: A BERT model for disambiguating abbreviations and contractions
Abbreviations and contractions are commonly found in text across different domains. For example, doctors' notes contain many contractions that can be personalized based on their choices. Existing spelling correction mode…
Spelling CorrectionEstimating Sparse Discrete Distributions Under Local Privacy and Communication Constraints
We consider the problem of estimating sparse discrete distributions under local differential privacy (LDP) and communication constraints. We characterize the sample complexity for sparse estimation under LDP constraints …
Dynamic industry uncertainty networks and the business cycle
We argue that uncertainty network structures extracted from option prices contain valuable information for business cycles. Classifying U.S. industries according to their contribution to system-related uncertainty across…
Recovering the Structural Observability of Composite Networks via Cartesian Product
Observability is a fundamental concept in system inference and estimation. This paper is focused on structural observability analysis of Cartesian product networks. Cartesian product networks emerge in variety of applica…