Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddings
In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into the class of polynomial systems. This embedding preserves input-output behavior and can suitably be extended to other neural DE architectures. We then use realization theory of polynomial systems to provide necessary conditions for an input-output map to be realizable by an ODE-LSTM and sufficient conditions for minimality of such systems. These results represent the first steps towards realization theory of recurrent neural ODE architectures, which is is expected be useful for model reduction and learning algorithm analysis of recurrent neural ODEs.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Dimension reduction in recurrent networks by canonicalization
Many recurrent neural network machine learning paradigms can be formulated using state-space representations. The classical notion of canonical state-space realization is adapted in this paper to accommodate semi-infinit…
Dimensionality ReductionSHiPPO: Recurrent Memory with Transported Polynomial Projections
HiPPO gives recurrent states memory semantics as coefficients of online polynomial projections, but in fixed channel coordinates. Modern selective SSMs, by contrast, rely on token-dependent control and channel interactio…
Interpretable Design of Reservoir Computing Networks using Realization Theory
The reservoir computing networks (RCNs) have been successfully employed as a tool in learning and complex decision-making tasks. Despite their efficiency and low training cost, practical applications of RCNs rely heavily…
Decision MakingPrimitive Recursion without Composition: Dynamical Characterizations, from Neural Networks to Polynomial ODEs
What do recurrent neural networks, polynomial ODEs, and discrete polynomial maps each bring to computation, and what do they lack? All three operate over the continuum--real-valued states evolved by real-valued dynamics-…
Approximating meta-heuristics with homotopic recurrent neural networks
Much combinatorial optimisation problems constitute a non-polynomial (NP) hard optimisation problem, i.e., they can not be solved in polynomial time. One such problem is finding the shortest route between two nodes on a …
Reinforcement LearningReinforcement Learning (RL)