Accelerating Neural Network Optimization Through an Automated Control Theory Lens
This paper studies the optimizer for accelerating the time-consuming deep network training through an automated control theory lens. We view the parameter update of a network as a feedback control process. It brings two contributions: First, we theoretically analyze the detailed intrinsic connections between deep network training and automatic control feedback system. Specifically, we demonstrate that the optimization process can be viewed as a Type I Second Order System in control field. Second, based on the math model of the equivalent system, we further design a proportional-integral-derivative algorithm type Controller with decoupled weight decay based on control theory to improve the training of deep neural networks. We conduct experiments both from a control theory lens through a phase locus verification and from a network training lens on several models, including CNNs, Transformers, MLPs, and on benchmark datasets. The results demonstrate the effectiveness of our Controller optimizer in both optimization speed and performance compared to SGD, PID Optimizer, Adam, AdamW and AdamP.
Code (0)
등록된 구현이 없습니다.
Tasks
MathMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Stochastic Resetting Accelerates Policy Convergence in Reinforcement Learning
Stochastic resetting, where a dynamical process is intermittently returned to a fixed reference state, has emerged as a powerful mechanism for optimizing first-passage properties. Existing theory largely treats static, n…
Reinforcement LearningContinuous ControlBayesian Co-navigation: Dynamic Designing of the Materials Digital Twins via Active Learning
Scientific advancement is universally based on the dynamic interplay between theoretical insights, modelling, and experimental discoveries. However, this feedback loop is often slow, including delayed community interacti…
Active LearningPAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization
Circuit discovery, which involves identifying sparse and task-relevant subnetworks in pre-trained language models, is a cornerstone of mechanistic interpretability. Automated Circuit Discovery (ACDC) has emerged as a piv…
Accelerating process control and optimization via machine learning: A review
Process control and optimization have been widely used to solve decision-making problems in chemical engineering applications. However, identifying and tuning the best solution algorithm is challenging and time-consuming…
Decision MakingReinforcement Learning Increases Wind Farm Power Production by Enabling Closed-Loop Collaborative Control
Traditional wind farm control operates each turbine independently to maximize individual power output. However, coordinated wake steering across the entire farm can substantially increase the combined wind farm energy pr…
Bayesian OptimizationReinforcement Learning (RL)