Learning Invariances in Neural Networks from Training Data
Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model should be invariant to a given augmentation. We show how to learn invariances by parameterizing a distribution over augmentations and optimizing the training loss simultaneously with respect to the network parameters and augmentation parameters. With this simple procedure we can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations, on training data alone. We show our approach is competitive with methods that are specialized to each task with the appropriate hard-coded invariances, without providing any prior knowledge of which invariance is needed.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationMolecular Property PredictionProperty PredictionSimilar Papers 제목 키워드 기반
Learning Invariances in Neural Networks
Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model sh…
image-classificationImage ClassificationMolecular Property PredictionProperty Prediction+1Characterizing the invariances of learning algorithms using category theory
Many learning algorithms have invariances: when their training data is transformed in certain ways, the function they learn transforms in a predictable manner. Here we formalize this notion using concepts from the mathem…
regressionDo Invariances in Deep Neural Networks Align with Human Perception?
An evaluation criterion for safe and trustworthy deep learning is how well the invariances captured by representations of deep neural networks (DNNs) are shared with humans. We identify challenges in measuring these inva…
Data AugmentationSelf-Supervised LearningA Closer Look at Invariances in Self-supervised Pre-training for 3D Vision
Self-supervised pre-training for 3D vision has drawn increasing research interest in recent years. In order to learn informative representations, a lot of previous works exploit invariances of 3D features, e.g., perspect…
Contrastive Learningobject-detectionObject DetectionSelf-Supervised Learning of Video-Induced Visual Invariances
We propose a general framework for self-supervised learning of transferable visual representations based on Video-Induced Visual Invariances (VIVI). We consider the implicit hierarchy present in the videos and make use o…
Image ClassificationSelf-Supervised LearningTransfer Learning