Optimization Dynamics of Equivariant and Augmented Neural Networks
We investigate the optimization of neural networks on symmetric data, and compare the strategy of constraining the architecture to be equivariant to that of using data augmentation. Our analysis reveals that that the relative geometry of the admissible and the equivariant layers, respectively, plays a key role. Under natural assumptions on the data, network, loss, and group of symmetries, we show that compatibility of the spaces of admissible layers and equivariant layers, in the sense that the corresponding orthogonal projections commute, implies that the sets of equivariant stationary points are identical for the two strategies. If the linear layers of the network also are given a unitary parametrization, the set of equivariant layers is even invariant under the gradient flow for augmented models. Our analysis however also reveals that even in the latter situation, stationary points may be unstable for augmented training although they are stable for the manifestly equivariant models.
Code (1)
Tasks
Data AugmentationSimilar Papers 제목 키워드 기반
Equivariant Neural Tangent Kernels
Little is known about the training dynamics of equivariant neural networks, in particular how it compares to data augmented training of their non-equivariant counterparts. Recently, neural tangent kernels (NTKs) have eme…
Data Augmentationimage-classificationImage ClassificationMedical Image Analysis+1Time-Equivariant Contrastive Video Representation Learning
We introduce a novel self-supervised contrastive learning method to learn representations from unlabelled videos. Existing approaches ignore the specifics of input distortions, e.g., by learning invariance to temporal tr…
Action RecognitionContrastive LearningRepresentation LearningRetrieval+1Flow Equivariant World Models: Memory for Partially Observed Dynamic Environments
Embodied systems experience the world as 'a symphony of flows': a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams an…
Rotation Equivariant Operators for Machine Learning on Scalar and Vector Fields
We develop theory and software for rotation equivariant operators on scalar and vector fields, with diverse applications in simulation, optimization and machine learning. Rotation equivariance (covariance) means all fiel…
BIG-bench Machine LearningTranslationEquivariance and Augmentation for Bayesian Neural Networks
Symmetries are important for many deep learning tasks, ranging from applications in the sciences to medical imaging. However, there is an ongoing debate about whether to impose symmetry constraints on the neural network …
Data Augmentation