G-TRACER: Expected Sharpness Optimization
We propose a new regularization scheme for the optimization of deep learning architectures, G-TRACER ("Geometric TRACE Ratio"), which promotes generalization by seeking flat minima, and has a sound theoretical basis as an approximation to a natural-gradient descent based optimization of a generalized Bayes objective. By augmenting the loss function with a TRACER, curvature-regularized optimizers (eg SGD-TRACER and Adam-TRACER) are simple to implement as modifications to existing optimizers and don't require extensive tuning. We show that the method converges to a neighborhood (depending on the regularization strength) of a local minimum of the unregularized objective, and demonstrate competitive performance on a number of benchmark computer vision and NLP datasets, with a particular focus on challenging low signal-to-noise ratio problems.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Zeroth-Order Sharpness-Aware Learning with Exponential Tilting
Classic zeroth-order optimization approaches typically optimize for a smoothed version of the original function, i.e., the expected objective under randomly perturbed model parameters. This can be interpreted as encourag…
Burden of intracerebral haemorrhage in Europe: forecasting incidence and mortality between 2019 and 2050
Summary Background Anticipating the burden of intracerebral haemorrhage is crucial for proactive management and building resilience against future health challenges. Prior forecasts are based on population demography a…
Model Generalization: A Sharpness Aware Optimization Perspective
Sharpness-Aware Minimization (SAM) and adaptive sharpness-aware minimization (ASAM) aim to improve the model generalization. And in this project, we proposed three experiments to valid their generalization from the sharp…
modelvalidRényi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization
Sharpness (of the loss minima) is widely believed to be a good indicator of generalization of neural networks. Unfortunately, the correlation between existing sharpness measures and generalization is not as strong as exp…
PET Tracer Separation Using Conditional Diffusion Transformer with Multi-latent Space Learning
In clinical practice, single-radiotracer positron emission tomography (PET) is commonly used for imaging. Although multi-tracer PET imaging can provide supplementary information of radiotracers that are sensitive to phys…
Computational Efficiency