Activation Functions for "A Feedforward Unitary Equivariant Neural Network"
In our previous work [Ma and Chan (2023)], we presented a feedforward unitary equivariant neural network. We proposed three distinct activation functions tailored for this network: a softsign function with a small residue, an identity function, and a Leaky ReLU function. While these functions demonstrated the desired equivariance properties, they limited the neural network's architecture. This short paper generalises these activation functions to a single functional form. This functional form represents a broad class of functions, maintains unitary equivariance, and offers greater flexibility for the design of equivariant neural networks.
Code (0)
등록된 구현이 없습니다.
Tasks
FormMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Feedforward Unitary Equivariant Neural Network
We devise a new type of feedforward neural network. It is equivariant with respect to the unitary group $U(n)$. The input and output can be vectors in $\mathbb{C}^n$ with arbitrary dimension $n$. No convolution layer is …
Scale Equivariant Graph Metanetworks
This paper pertains to an emerging machine learning paradigm: learning higher-order functions, i.e. functions whose inputs are functions themselves, $\textit{particularly when these inputs are Neural Networks (NNs)}$. Wi…
Inductive BiasData-Driven Learning of Feedforward Neural Networks with Different Activation Functions
This work contributes to the development of a new data-driven method (D-DM) of feedforward neural networks (FNNs) learning. This method was proposed recently as a way of improving randomized learning of FNNs by adjusting…
Precoder Learning by Leveraging Unitary Equivariance Property
Incorporating mathematical properties of a wireless policy to be learned into the design of deep neural networks (DNNs) is effective for enhancing learning efficiency. Multi-user precoding policy in multi-antenna system,…
Bounds on the Approximation Power of Feedforward Neural Networks
The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error …