Efficient Finite Initialization for Tensorized Neural Networks
We present a novel method for initializing layers of tensorized neural networks in a way that avoids the explosion of the parameters of the matrix it emulates. The method is intended for layers with a high number of nodes in which there is a connection to the input or output of all or most of the nodes, we cannot or do not want to store/calculate all the elements of the represented layer and they follow a smooth distribution. This method is equally applicable to normalize general tensor networks in which we want to avoid overflows. The core of this method is the use of the Frobenius norm and the partial lineal entrywise norm of reduced forms of the layer in an iterative partial form, so that it has to be finite and within a certain range. These norms are efficient to compute, fully or partially for most cases of interest. In addition, the method benefits from the reuse of intermediate calculations. We apply the method to different layers and check its performance. We create a Python function to run it on an arbitrary layer, available in a Jupyter Notebook in the i3BQuantum repository: https://github.com/i3BQuantumTeam/Q4Real/blob/e07c827651ef16bcf74590ab965ea3985143f891/Quantum-Inspired%20Variational%20Methods/TN_Normalizer.ipynb
Code (0)
등록된 구현이 없습니다.
Tasks
Tensor NetworksSimilar Papers 제목 키워드 기반
Mitigating Bias in Locally Constrained Decoding via Tractable Proposals
Generations from large language models often fail to conform to desired constraints such as JSON schema. Existing locally constrained decoding (LCD) approaches enforce constraints by myopically masking out next tokens, r…
Tensorized Optical Multimodal Fusion Network
We propose the first tensorized optical multimodal fusion network architecture with a self-attention mechanism and low-rank tensor fusion. Simulation results show $51.3 \times$ less hardware requirement and $3.7\times 10…
Rademacher Random Projections with Tensor Networks
Random projection (RP) have recently emerged as popular techniques in the machine learning community for their ability in reducing the dimension of very high-dimensional tensors. Following the work in [30], we consider a…
Tensor NetworksUNIT: Unifying Tensorized Instruction Compilation
Because of the increasing demand for computation in DNN, researchers develope both hardware and software mechanisms to reduce the compute and memory burden. A widely adopted approach is to use mixed precision data types.…
CPUReducing Computational Costs in Sentiment Analysis: Tensorized Recurrent Networks vs. Recurrent Networks
Anticipating audience reaction towards a certain text is integral to several facets of society ranging from politics, research, and commercial industries. Sentiment analysis (SA) is a useful natural language processing (…
Sentiment Analysis