Neural Arithmetic Logic Units
Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training. To encourage more systematic numerical extrapolation, we propose an architecture that represents numerical quantities as linear activations which are manipulated using primitive arithmetic operators, controlled by learned gates. We call this module a neural arithmetic logic unit (NALU), by analogy to the arithmetic logic unit in traditional processors. Experiments show that NALU-enhanced neural networks can learn to track time, perform arithmetic over images of numbers, translate numerical language into real-valued scalars, execute computer code, and count objects in images. In contrast to conventional architectures, we obtain substantially better generalization both inside and outside of the range of numerical values encountered during training, often extrapolating orders of magnitude beyond trained numerical ranges.
Code (36)
Similar Papers 제목 키워드 기반
Neural Arithmetic Units
Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers. The lack of inductive bias for arithmetic operations leaves neural networks without the under…
Inductive BiasConvolutional Feature Extraction and Neural Arithmetic Logic Units for Stock Prediction
Stock prediction is a topic undergoing intense study for many years. Finance experts and mathematicians have been working on a way to predict the future stock price so as to decide to buy the stock or sell it to make pro…
Sentiment AnalysisStock PredictionAlgoritmos de minería de datos en la industria sanitaria
In this paper, we review data mining approaches for health applications. Our focus is on hardware-centric approaches. Modern computers consist of multiple processors, each equipped with multiple cores, each with a set of…
EEGElectroencephalogram (EEG)Systematically designing better instance counting models on cell images with Neural Arithmetic Logic Units
The big problem for neural network models which are trained to count instances is that whenever test range goes high training range generalization error increases i.e. they are not good generalizers outside training rang…
Data AugmentationregressionGENIAL: Generative Design Space Exploration via Network Inversion for Low Power Algorithmic Logic Units
As AI workloads proliferate, optimizing arithmetic units is becoming increasingly important for reducing the footprint of digital systems. Conventional design flows, which often rely on manual or heuristic-based optimiza…