Quantized Approximately Orthogonal Recurrent Neural Networks
In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the copy-task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices.In this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient models achieve results similar to state-of-the-art full-precision ORNN, LSTM and FastRNN on a variety of standard benchmarks, even with 4-bits quantization.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationTime SeriesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Coordinate descent on the orthogonal group for recurrent neural network training
We propose to use stochastic Riemannian coordinate descent on the orthogonal group for recurrent neural network training. The algorithm rotates successively two columns of the recurrent matrix, an operation that can be e…
QCM-SGM+: Improved Quantized Compressed Sensing With Score-Based Generative Models
In practical compressed sensing (CS), the obtained measurements typically necessitate quantization to a limited number of bits prior to transmission or storage. This nonlinear quantization process poses significant recov…
Bayesian Inferencecompressed sensingQuantizationWhen Quantization Breaks Memory: Recurrent-State Write-Back in Low-Precision Temporal Inference
Quantization is widely used to reduce the computational and memory demands of neural-network inference. In recurrent networks, however, the quantized state is stored and returned at the next time step, so the rule used t…
Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networks
The deployment of quantized neural networks on edge devices, combined with privacy regulations like GDPR, creates an urgent need for machine unlearning in quantized models. However, existing methods face critical challen…
Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformation
In recent years, using orthogonal matrices has been shown to be a promising approach in improving Recurrent Neural Networks (RNNs) with training, stability, and convergence, particularly, to control gradients. While Gate…