Multilinear Operator Networks
Despite the remarkable capabilities of deep neural networks in image recognition, the dependence on activation functions remains a largely unexplored area and has yet to be eliminated. On the other hand, Polynomial Networks is a class of models that does not require activation functions, but have yet to perform on par with modern architectures. In this work, we aim close this gap and propose MONet, which relies solely on multilinear operators. The core layer of MONet, called Mu-Layer, captures multiplicative interactions of the elements of the input token. MONet captures high-degree interactions of the input elements and we demonstrate the efficacy of our approach on a series of image recognition and scientific computing benchmarks. The proposed model outperforms prior polynomial networks and performs on par with modern architectures. We believe that MONet can inspire further research on models that use entirely multilinear operations.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A New Approach to Multilinear Dynamical Systems and Control
The current paper presents a new approach to multilinear dynamical systems analysis and control. The approach is based upon recent developments in tensor decompositions and a newly defined algebra of circulants. In parti…
Low-Rank Adaptation in Multilinear Operator Networks for Security-Preserving Incremental Learning
In security-sensitive fields, data should be encrypted to protect against unauthorized access and maintain confidentiality throughout processing. However, traditional networks like ViTs and CNNs return different resu…
Incremental LearningNonlinear tensor product approximation of functions
We are interested in approximation of a multivariate function $f(x_1,\dots,x_d)$ by linear combinations of products $u^1(x_1)\cdots u^d(x_d)$ of univariate functions $u^i(x_i)$, $i=1,\dots,d$. In the case $d=2$ it is a c…
Enabling Real-Time Training of a Wildfire-to-Smoke Map with Multilinear Operators
Wildfires are a major producer of fine particulate matter, impacting human health and the electrical grid. Accurately forecasting smoke impacts over long time scales incorporates fuel treatment strategies, natural fuel s…
An unsupervised capacity identification approach based on Sobol' indices
In many ranking problems, some particular aspects of the addressed situation should be taken into account in the aggregation process. An example is the presence of correlations between criteria, which may introduce bias …