Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks

2019-12-01 · NeurIPS 2019 12 · Kohei Hayashi, Taiki Yamaguchi, Yohei Sugawara, Shin-ichi Maeda

Tensor decomposition methods are widely used for model compression and fast inference in convolutional neural networks (CNNs). Although many decompositions are conceivable, only CP decomposition and a few others have been applied in practice, and no extensive comparisons have been made between available methods. Previous studies have not determined how many decompositions are available, nor which of them is optimal. In this study, we first characterize a decomposition class specific to CNNs by adopting a flexible graphical notation. The class includes such well-known CNN modules as depthwise separable convolution layers and bottleneck layers, but also previously unknown modules with nonlinear activations. We also experimentally compare the tradeoff between prediction accuracy and time/space complexity for modules found by enumerating all possible decompositions, or by using a neural architecture search. We find some nonlinear decompositions outperform existing ones.

📄 PDF Abstract BibTeX

Code (1)

pfnet-research/einconv 공식 구현 pytorch

Tasks

Model Compression Neural Architecture SearchTensor Decomposition

Methods 이 논문이 사용한 방법론

Depthwise Convolution Depthwise Convolution is a type of convolution where we apply a single convolutional filter for each input channel. In the regular 2D…

Pointwise Convolution Pointwise Convolution is a type of convolution that uses a 1x1 kernel: a kernel that iterates through every single point. This…

Depthwise Separable Convolution While standard convolution performs the channelwise and spatial-wise computation in one step, Depthwise Separable Convolution …

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Einconv: Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks

2019-08-13 · Kohei Hayashi, Taiki Yamaguchi, Yohei Sugawara, Shin-ichi Maeda

Model Compression Neural Architecture Search Tensor Decomposition

Model based Multi-agent Reinforcement Learning with Tensor Decompositions

2021-10-27 · Pascal Van Der Vaart, Anuj Mahajan, Shimon Whiteson

A challenge in multi-agent reinforcement learning is to be able to generalize over intractable state-action spaces. Inspired from Tesseract [Mahajan et al., 2021], this position paper investigates generalisation in state…

Model-based Reinforcement Learning Multi-agent Reinforcement Learning Position reinforcement-learning+2

Tensor network compressibility of convolutional models

2024-03-21 · Sukhbinder Singh, Saeed S. Jahromi, Roman Orus

Convolutional neural networks (CNNs) are one of the most widely used neural network architectures, showcasing state-of-the-art performance in computer vision tasks. Although larger CNNs generally exhibit higher accuracy,…

image-classification Image Classification

Tensor-Dictionary Learning with Deep Kruskal-Factor Analysis

2016-12-08 · Andrew Stevens, Yunchen Pu, Yannan Sun, Greg Spell 외

A multi-way factor analysis model is introduced for tensor-variate data of any order. Each data item is represented as a (sparse) sum of Kruskal decompositions, a Kruskal-factor analysis (KFA). KFA is nonparametric and c…

Denoising Dictionary Learning General Classification image-classification+1

Multidimensional Data Analysis Based on Block Convolutional Tensor Decomposition

2023-08-03 · Mahdi Molavi, Mansoor Rezghi, Tayyebeh Saeedi

Tensor decompositions are powerful tools for analyzing multi-dimensional data in their original format. Besides tensor decompositions like Tucker and CP, Tensor SVD (t-SVD) which is based on the t-product of tensors is a…

Tensor Decomposition