Transformation of ReLU-based recurrent neural networks from discrete-time to continuous-time
Recurrent neural networks (RNN) as used in machine learning are commonly formulated in discrete time, i.e. as recursive maps. This brings a lot of advantages for training models on data, e.g. for the purpose of time series prediction or dynamical systems identification, as powerful and efficient inference algorithms exist for discrete time systems and numerical integration of differential equations is not necessary. On the other hand, mathematical analysis of dynamical systems inferred from data is often more convenient and enables additional insights if these are formulated in continuous time, i.e. as systems of ordinary (or partial) differential equations (ODE). Here we show how to perform such a translation from discrete to continuous time for a particular class of ReLU-based RNN. We prove three theorems on the mathematical equivalence between the discrete and continuous time formulations under a variety of conditions, and illustrate how to use our mathematical results on different machine learning and nonlinear dynamical systems examples.
Code (1)
Tasks
BIG-bench Machine LearningNumerical IntegrationTime SeriesTime Series AnalysisTime Series PredictionTranslationSimilar Papers 제목 키워드 기반
Continuous Depth Recurrent Neural Differential Equations
Recurrent neural networks (RNNs) have brought a lot of advancements in sequence labeling tasks and sequence data. However, their effectiveness is limited when the observations in the sequence are irregularly sampled, whe…
The Symmetries of Three-Layer ReLU Networks
We develop a framework for analyzing parameter symmetries in deep ReLU networks and obtain a complete characterization of the generic parameter fibers for three-layer bottleneck architectures. Our approach provides expli…
Stability and Performance Analysis of Discrete-Time ReLU Recurrent Neural Networks
This paper presents sufficient conditions for the stability and $\ell_2$-gain performance of recurrent neural networks (RNNs) with ReLU activation functions. These conditions are derived by combining Lyapunov/dissipativi…
Primitive Recursion without Composition: Dynamical Characterizations, from Neural Networks to Polynomial ODEs
What do recurrent neural networks, polynomial ODEs, and discrete polynomial maps each bring to computation, and what do they lack? All three operate over the continuum--real-valued states evolved by real-valued dynamics-…
Gates Are Not What You Need in RNNs
Recurrent neural networks have flourished in many areas. Consequently, we can see new RNN cells being developed continuously, usually by creating or using gates in a new, original way. But what if we told you that gates …
Language ModelingLanguage ModellingMusic ModelingSentiment Analysis