Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers
Neural network-based methods have emerged as powerful tools for solving partial differential equations (PDEs) in scientific and engineering applications, particularly when handling complex domains or incorporating empirical data. These methods leverage neural networks as basis functions to approximate PDE solutions. However, training such networks can be challenging, often resulting in limited accuracy. In this paper, we investigate the training dynamics of neural network-based PDE solvers with a focus on the impact of initialization techniques. We assess training difficulty by analyzing the eigenvalue distribution of the kernel and apply the concept of effective rank to quantify this difficulty, where a larger effective rank correlates with faster convergence of the training error. Building upon this, we discover through theoretical analysis and numerical experiments that two initialization techniques, partition of unity (PoU) and variance scaling (VS), enhance the effective rank, thereby accelerating the convergence of training error. Furthermore, comprehensive experiments using popular PDE-solving frameworks, such as PINN, Deep Ritz, and the operator learning framework DeepOnet, confirm that these initialization techniques consistently speed up convergence, in line with our theoretical findings.
Code (0)
등록된 구현이 없습니다.
Tasks
Operator learningUnityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
NSPOD: Accelerating Krylov solvers via DeepONet-learned POD subspaces
The convergence of Krylov-based linear iterative solvers applied to parametric partial differential equations (PDEs) is often highly sensitive to the domain, its discretization, the location/values of the applied Dirichl…
Learning Provably Improves the Convergence of Gradient Descent
Learn to Optimize (L2O) trains deep neural network based solvers for optimization, achieving success in accelerating convex problems and improving non-convex solutions. However, L2O lacks rigorous theoretical backing for…
SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning
Training large language models with reinforcement learning (RL) against verifiable rewards significantly enhances their reasoning abilities, yet remains computationally expensive due to inefficient uniform prompt samplin…
Reinforcement Learning (RL)Predictive Batch Scheduling: Accelerating Language Model Training Through Loss-Aware Sample Prioritization
We introduce Predictive Batch Scheduling (PBS), a novel training optimization technique that accelerates language model convergence by dynamically prioritizing high-loss samples during batch construction. Unlike curricul…
Accelerating Quasi-Static Time Series Simulations with Foundation Models
Quasi-static time series (QSTS) simulations have great potential for evaluating the grid's ability to accommodate the large-scale integration of distributed energy resources. However, as grids expand and operate closer t…
Time Series