Feed-Forward Neural Networks Need Inductive Bias to Learn Equality Relations
Basic binary relations such as equality and inequality are fundamental to relational data structures. Neural networks should learn such relations and generalise to new unseen data. We show in this study, however, that this generalisation fails with standard feed-forward networks on binary vectors. Even when trained with maximal training data, standard networks do not reliably detect equality.We introduce differential rectifier (DR) units that we add to the network in different configurations. The DR units create an inductive bias in the networks, so that they do learn to generalise, even from small numbers of examples and we have not found any negative effect of their inclusion in the network. Given the fundamental nature of these relations, we hypothesize that feed-forward neural network learning benefits from inductive bias in other relations as well. Consequently, the further development of suitable inductive biases will be beneficial to many tasks in relational learning with neural networks.
Code (0)
등록된 구현이 없습니다.
Tasks
Inductive BiasRelational ReasoningSimilar Papers 제목 키워드 기반
In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix…
Inductive BiasPUFFIN: A Path-Unifying Feed-Forward Interfaced Network for Vapor Pressure Prediction
Accurately predicting vapor pressure is vital for various industrial and environmental applications. However, obtaining accurate measurements for all compounds of interest is not possible due to the resource and labor in…
Inductive BiasPredictionTransfer LearningGANORCON: Are Generative Models Useful for Few-shot Segmentation?
Advances in generative modeling based on GANs has motivated the community to find their use beyond image generation and editing tasks. In particular, several recent works have shown that GAN representations can be re-pur…
Contrastive LearningImage GenerationSelf-Supervised LearningEncoding the Euler Characteristic Transform
The Euler Characteristic Curve (ECC) records the Euler characteristic of a linearly embedded cell complex as a function of filtration height in a given direction, and the Euler Characteristic Transform (ECT) is the injec…
Point CloudsEvery Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity
We show that, in a precise sense, a broad class of feedforward neural networks learn (have finite sample complexity) in the PAC model: every fixed finite feedforward architecture whose layers are definable in an o-minima…