A Group-Theoretic Framework for Data Augmentation
Data augmentation is a widely used trick when training deep neural networks: in addition to the original data, properly transformed data are also added to the training set. However, to the best of our knowledge, a clear mathematical framework to explain the performance benefits of data augmentation is not available. In this paper, we develop such a theoretical framework. We show data augmentation is equivalent to an averaging operation over the orbits of a certain group that keeps the data distribution approximately invariant. We prove that it leads to variance reduction. We study empirical risk minimization, and the examples of exponential families, linear regression, and certain two-layer neural networks. We also discuss how data augmentation could be used in problems with symmetry where other approaches are prevalent, such as in cryo-electron microscopy (cryo-EM).
Code (1)
Tasks
Data AugmentationImage ClassificationSimilar Papers 제목 키워드 기반
The geometry of invariant learning: an information-theoretic analysis of data augmentation and generalization
Data augmentation is one of the most widely used techniques to improve generalization in modern machine learning, often justified by its ability to promote invariance to label-irrelevant transformations. However, its the…
Data AugmentationData Augmentation: A Fourier Analysis Perspective
Data augmentation is a simple and model-agnostic approach for exploiting known invariances in learning problems. Given a group acting on the input space, one augments the training set with transformed copies of each samp…
Data AugmentationModel Patching: Closing the Subgroup Performance Gap with Data Augmentation
Classifiers in machine learning are often brittle when deployed. Particularly concerning are models with inconsistent performance on specific subgroups of a class, e.g., exhibiting disparities in skin cancer classificati…
Cancer ClassificationData AugmentationSkin Cancer ClassificationRestrained Generative Adversarial Network against Overfitting in Numeric Data Augmentation
In recent studies, Generative Adversarial Network (GAN) is one of the popular schemes to augment the image dataset. However, in our study we find the generator G in the GAN fails to generate numerical data in lower-dimen…
Data AugmentationGenerative Adversarial NetworkHow Tempering Fixes Data Augmentation in Bayesian Neural Networks
While Bayesian neural networks (BNNs) provide a sound and principled alternative to standard neural networks, an artificial sharpening of the posterior usually needs to be applied to reach comparable performance. This is…
Data Augmentation