SEROAISE: Advancing ROA Estimation for ReLU and PWA Dynamics through Estimating Certified Invariant Sets
This paper presents a novel framework for constructing the Region of Attraction (RoA) for dynamics derived either from Piecewise Affine (PWA) functions or from Neural Networks (NNs) with Rectified Linear Units (ReLU) activation function. This method, described as Sequential Estimation of RoA based on Invariant Set Estimation (SEROAISE), computes a Lyapunov-like PWA function over a certified PWA invariant set. While traditional approaches search for Lyapunov functions by enforcing Lyapunov conditions over pre-selected domains, this framework enforces Lyapunov-like conditions over a certified invariant subset obtained using the Iterative Invariant Set Estimator(IISE). Compared to the state-of-the-art, IISE provides systematically larger certified invariant sets. In order to find a larger invariant subset, the IISE utilizes a novel concept known as the Non-Uniform Growth of Invariant Set (NUGIS). A number of examples illustrating the efficacy of the proposed methods are provided, including dynamical systems derived from learning algorithms. The implementation is publicly available at: https://github.com/PouyaSamanipour/SEROAISE.git.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networks
Understanding implicit bias of gradient descent for generalization capability of ReLU networks has been an important research topic in machine learning research. Unfortunately, even for a single ReLU neuron trained with …
Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
Among many mysteries behind the success of deep networks lies the exceptional discriminative power of their learned representations as manifested by the intriguing Neural Collapse (NC) phenomenon, where simple feature st…
On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting
Deep learning empirically achieves high performance in many applications, but its training dynamics has not been fully understood theoretically. In this paper, we explore theoretical analysis on training two-layer ReLU n…
When Are Bias-Free ReLU Networks Effectively Linear Networks?
We investigate the implications of removing bias in ReLU networks regarding their expressivity and learning dynamics. We first show that two-layer bias-free ReLU networks have limited expressivity: the only odd function …
Magnitude and Angle Dynamics in Training Single ReLU Neurons
To understand learning the dynamics of deep ReLU networks, we investigate the dynamic system of gradient flow $w(t)$ by decomposing it to magnitude $w(t)$ and angle $\phi(t):= \pi - \theta(t) $ components. In particular,…