Learning Invariances in Neural Networks
Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model should be invariant to a given symmetry group. We show how to \emph{learn} invariances and equivariances by parameterizing a distribution over augmentations and optimizing the training loss simultaneously with respect to the network parameters and augmentation parameters. With this simple procedure we can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations, on training data alone.
Code (1)
Tasks
image-classificationImage ClassificationMolecular Property PredictionProperty PredictionregressionSimilar Papers 제목 키워드 기반
Learning Invariances in Neural Networks from Training Data
Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model sh…
image-classificationImage ClassificationMolecular Property PredictionProperty PredictionDo Invariances in Deep Neural Networks Align with Human Perception?
An evaluation criterion for safe and trustworthy deep learning is how well the invariances captured by representations of deep neural networks (DNNs) are shared with humans. We identify challenges in measuring these inva…
Data AugmentationSelf-Supervised LearningCharacterizing the invariances of learning algorithms using category theory
Many learning algorithms have invariances: when their training data is transformed in certain ways, the function they learn transforms in a predictable manner. Here we formalize this notion using concepts from the mathem…
regressionLearning to Exploit Invariances in Clinical Time-Series Data using Sequence Transformer Networks
Recently, researchers have started applying convolutional neural networks (CNNs) with one-dimensional convolutions to clinical tasks involving time-series data. This is due, in part, to their computational efficiency, re…
Computational EfficiencyDynamic Time WarpingTime SeriesTime Series AnalysisInvariances of random fields paths, with applications in Gaussian Process Regression
We study pathwise invariances of centred random fields that can be controlled through the covariance. A result involving composition operators is obtained in second-order settings, and we show that various path propertie…
regression