Data-induced multiscale losses and efficient multirate gradient descent schemes
This paper investigates the impact of multiscale data on machine learning algorithms, particularly in the context of deep learning. A dataset is multiscale if its distribution shows large variations in scale across different directions. This paper reveals multiscale structures in the loss landscape, including its gradients and Hessians inherited from the data. Correspondingly, it introduces a novel gradient descent approach, drawing inspiration from multiscale algorithms used in scientific computing. This approach seeks to transcend empirical learning rate selection, offering a more systematic, data-informed strategy to enhance training efficiency, especially in the later stages.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling
Many particle-based Bayesian inference methods use a single global step size for all parts of the update. In Stein variational gradient descent (SVGD), however, each update combines two qualitatively different effects: a…
Bayesian InferenceFrom Simple to Complex: Curriculum-Guided Physics-Informed Neural Networks via Gaussian Mixture Models
Physics-informed neural networks (PINNs) offer a mesh-free framework for solving partial differential equations (PDEs), yet training often suffers from gradient pathologies, spectral bias, and poor convergence, especiall…
Conflict-Aware Harmonized Rotational Gradient for Multiscale Kinetic Regimes
In this paper, we propose a harmonized rotational gradient method, termed HRGrad, for simultaneously tackling multiscale time-dependent kinetic problems with varying small parameters. These parameters exhibit asymptotic …
Multi-Task LearningMatrix Pencil-Based Analysis of Multirate Simulation Schemes
This paper focuses on multirate time-domain simulations of power system models. It proposes a matrix pencil-based approach to evaluate the spurious numerical deformation introduced into power system dynamics by a given m…
Lifted Frequency-Domain Identification of Closed-Loop Multirate Systems: Applied to Dual-Stage Actuator Hard Disk Drives
Frequency-domain representations are crucial for the design and performance evaluation of controllers in multirate systems, specifically to address intersample performance. The aim of this paper is to develop an effectiv…