DADA: Differentiable Automatic Data Augmentation

2020-03-08 · ECCV 2020 8 · Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M. Robertson, Yongxin Yang

Data augmentation (DA) techniques aim to increase data variability, and thus train deep networks with better generalisation. The pioneering AutoAugment automated the search for optimal DA policies with reinforcement learning. However, AutoAugment is extremely computationally expensive, limiting its wide applicability. Followup works such as Population Based Augmentation (PBA) and Fast AutoAugment improved efficiency, but their optimization speed remains a bottleneck. In this paper, we propose Differentiable Automatic Data Augmentation (DADA) which dramatically reduces the cost. DADA relaxes the discrete DA policy selection to a differentiable optimization problem via Gumbel-Softmax. In addition, we introduce an unbiased gradient estimator, RELAX, leading to an efficient and effective one-pass optimization strategy to learn an efficient and accurate DA policy. We conduct extensive experiments on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. Furthermore, we demonstrate the value of Auto DA in pre-training for downstream detection problems. Results show our DADA is at least one order of magnitude faster than the state-of-the-art while achieving very comparable accuracy. The code is available at https://github.com/VDIGPKU/DADA.

📄 PDF Abstract BibTeX arXiv:2003.03780

Code (1)

VDIGPKU/DADA 공식 구현 pytorch

Tasks

Data Augmentation Reinforcement Learning

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Population Based Training Population Based Training, or PBT, is an optimization method for finding parameters and hyperparameters, and extends upon parallel search methods and sequential…

Population Based Augmentation Population Based Augmentation, or PBA, is a data augmentation strategy (PBA), which generates nonstationary augmentation policy schedules instead of a fixed augmentation…

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Fast AutoAugment Fast AutoAugment is an image data augmentation algorithm that finds effective augmentation policies via a search strategy based on density matching, motivated by Bayesian DA.…

AutoAugment 설명 없음

DADA: Deep Adversarial Data Augmentation for Extremely Low Data Regime Classification

2018-08-29 · Xiaofeng Zhang, Zhangyang Wang, Dong Liu, Qing Ling

Deep learning has revolutionized the performance of classification, but meanwhile demands sufficient labeled data for training. Given insufficient data, while many techniques have been developed to help combat overfittin…

Data Augmentation General Classification Generative Adversarial Network Transfer Learning

CADDA: Class-wise Automatic Differentiable Data Augmentation for EEG Signals

2021-06-25 · ICLR 2022 4 · Cédric Rommel, Thomas Moreau, Joseph Paillard, Alexandre Gramfort

Data augmentation is a key element of deep learning pipelines, as it informs the network during training about transformations of the input data that keep the label unchanged. Manually finding adequate augmentation metho…

Data Augmentation EEG Electroencephalogram (EEG)

Automatic Data Augmentation for 3D Medical Image Segmentation

2020-10-07 · Ju Xu, Mengzhang Li, Zhanxing Zhu

Data augmentation is an effective and universal technique for improving generalization performance of deep neural networks. It could enrich diversity of training samples that is essential in medical image segmentation ta…

Data Augmentation Diversity Image Segmentation Medical Image Segmentation+1

LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics

2024-10-21 · Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, Dan Alistarh

We introduce LDAdam, a memory-efficient optimizer for training large models, that performs adaptive optimization steps within lower dimensional subspaces, while consistently exploring the full parameter space during trai…

Efficient Adaptive Federated Optimization

2024-10-10 · Su Hyeong Lee, Sidharth Sharma, Manzil Zaheer, Tian Li

Adaptive optimization is critical in federated learning, where enabling adaptivity on both the server and client sides has proven essential for achieving optimal performance. However, the scalability of such jointly adap…

Federated Learning