State Derivative Normalization for Continuous-Time Deep Neural Networks
The importance of proper data normalization for deep neural networks is well known. However, in continuous-time state-space model estimation, it has been observed that improper normalization of either the hidden state or hidden state derivative of the model estimate, or even of the time interval can lead to numerical and optimization challenges with deep learning based methods. This results in a reduced model quality. In this contribution, we show that these three normalization tasks are inherently coupled. Due to the existence of this coupling, we propose a solution to all three normalization challenges by introducing a normalization constant at the state derivative level. We show that the appropriate choice of the normalization constant is related to the dynamics of the to-be-identified system and we derive multiple methods of obtaining an effective normalization constant. We compare and discuss all the normalization strategies on a benchmark problem based on experimental data from a cascaded tanks system and compare our results with other methods of the identification literature.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Continuous-time identification of dynamic state-space models by deep subspace encoding
Continuous-time (CT) modeling has proven to provide improved sample efficiency and interpretability in learning the dynamical behavior of physical systems compared to discrete-time (DT) models. However, even with numerou…
State Space ModelsStableGrad: Backward Scale Control without Batch Normalization
Training very deep neural networks requires controlling the propagation of magnitudes across depth. Without such control, activations and gradients may vanish, explode, or enter unstable regimes that make optimization fa…
Higher-order Derivatives of Weighted Finite-state Machines
Weighted finite-state machines are a fundamental building block of NLP systems. They have withstood the test of time -- from their early use in noisy channel models in the 1990s up to modern-day neurally parameterized co…
Neural Stochastic Contraction Metrics for Learning-based Control and Estimation
We present Neural Stochastic Contraction Metrics (NSCM), a new design framework for provably-stable robust control and estimation for a class of stochastic nonlinear systems. It uses a spectrally-normalized deep neural n…
Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation
We introduce PACE, a backpropagation-free continual test-time adaptation system that directly optimizes the affine parameters of normalization layers. Existing derivative-free approaches struggle to balance runtime effic…
Test-time Adaptation