Bayesian Sparsification Methods for Deep Complex-valued Networks

2020-03-25 · Ivan Nazarov, Evgeny Burnaev

With continual miniaturization ever more applications of deep learning can be found in embedded systems, where it is common to encounter data with natural complex domain representation. To this end we extend Sparse Variational Dropout to complex-valued neural networks and verify the proposed Bayesian technique by conducting a large numerical study of the performance-compression trade-off of C-valued networks on two tasks: image recognition on MNIST-like and CIFAR10 datasets and music transcription on MusicNet. We replicate the state-of-the-art result by Trabelsi et al. [2018] on MusicNet with a complex-valued network compressed by 50-100x at a small performance penalty.

📄 PDF Abstract BibTeX arXiv:2003.11413

Code (1)

ivannz/complex_paper 공식 구현 pytorch

Tasks

Music Transcription

Methods 이 논문이 사용한 방법론

Variational Dropout Variational Dropout is a regularization technique based on dropout, but uses a variational inference grounded approach. In…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Bayesian Sparsification of Deep C-valued Networks

2020-01-01 · ICML 2020 1 · Ivan Nazarov, Evgeny Burnaev

With continual miniaturization ever more applications of deep learning can be found in embedded systems, where it is common to encounter data with natural representation in the complex domain. To this end we extend Spars…

Music Transcription

Bayesian sparsification for deep neural networks with Bayesian model reduction

2023-09-21 · Dimitrije Marković, Karl J. Friston, Stefan J. Kiebel

Deep learning's immense capabilities are often constrained by the complexity of its models, leading to an increasing demand for effective sparsification techniques. Bayesian sparsification for deep learning emerges as a …

Deep Learning Variational Inference

Spectral Complexity-scaled Generalization Bound of Complex-valued Neural Networks

2021-12-07 · Haowen Chen, Fengxiang He, Shiye Lei, DaCheng Tao

Complex-valued neural networks (CVNNs) have been widely applied to various fields, especially signal processing and image recognition. However, few works focus on the generalization of CVNNs, albeit it is vital to ensure…

LEMMA

Novel Gradient Sparsification Algorithm via Bayesian Inference

2024-09-23 · Ali Bereyhi, Ben Liang, Gary Boudreau, Ali Afana

Error accumulation is an essential component of the Top-$k$ sparsification method in distributed gradient descent. It implicitly scales the learning rate and prevents the slow-down of lateral movement, but it can also de…

Bayesian Inference

Bayesian Sparsification of Gated Recurrent Neural Networks

2018-12-12 · NIPS Workshop CDNNRIA 2018 · Ekaterina Lobacheva, Nadezhda Chirkova, Dmitry Vetrov

Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop this approach for gated recurrent archit…