Spectral Tensor Train Parameterization of Deep Learning Layers
We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optimization problems, leading to better models and stability of optimization. We start by looking at the compact SVD parameterization of weight matrices and identifying redundancy sources in the parameterization. We further apply the Tensor Train (TT) decomposition to the compact SVD components, and propose a non-redundant differentiable parameterization of fixed TT-rank tensor manifolds, termed the Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of neural network compression in the image classification setting and both compression and improved training stability in the generative adversarial training setting.
Code (1)
Tasks
Deep Learningimage-classificationImage ClassificationImage GenerationNeural Network CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fast and Structured Block-Term Tensor Decomposition For Hyperspectral Unmixing
The block-term tensor decomposition model with multilinear rank-$(L_r,L_r,1)$ terms (or, the "LL1 tensor decomposition" in short) offers a valuable alternative for hyperspectral unmixing (HU) under the linear mixture mod…
Hyperspectral UnmixingTensor DecompositionExtending $μ$P: Spectral Conditions for Feature Learning Across Optimizers
Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance of these optimization routines is highly …
Zero-Shot LearningFUTON: Fourier Tensor Network for Implicit Neural Representations
Implicit neural representations (INRs) have emerged as powerful tools for encoding signals, yet dominant MLP-based designs often suffer from slow convergence, overfitting to noise, and poor extrapolation. We introduce FU…
Image DenoisingTransformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks
Achieving efficient and robust multi-channel data learning is a challenging task in data science. By exploiting low-rankness in the transformed domain, i.e., transformed low-rankness, tensor Singular Value Decomposition …
Generalization BoundsTensor-on-Tensor Regression: Riemannian Optimization, Over-parameterization, Statistical-computational Gap, and Their Interplay
We study the tensor-on-tensor regression, where the goal is to connect tensor responses to tensor covariates with a low Tucker rank parameter tensor/matrix without the prior knowledge of its intrinsic rank. We propose th…
regressionRiemannian optimization