Accelerating AI Performance using Anderson Extrapolation on GPUs
We present a novel approach for accelerating AI performance by leveraging Anderson extrapolation, a vector-to-vector mapping technique based on a window of historical iterations. By identifying the crossover point (Fig. 1) where a mixing penalty is incurred, the method focuses on reducing iterations to convergence, with fewer more compute-intensive but generally cacheable iterations, balancing speed and memory usage with accuracy and algorithmic stability, respectively. We demonstrate significant improvements, in both training and inference, motivated by scalability and efficiency extensions to the realm of high-performance computing (HPC).
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Constructing artificial life and materials scientists with accelerated AI using Deep AndersoNN
Deep AndersoNN accelerates AI by exploiting the continuum limit as the number of explicit layers in a neural network approaches infinity and can be taken as a single implicit layer, known as a deep equilibrium model. Sol…
Artificial LifeInterpolatron: Interpolation or Extrapolation Schemes to Accelerate Optimization for Deep Neural Networks
In this paper we explore acceleration techniques for large scale nonconvex optimization problems with special focuses on deep neural networks. The extrapolation scheme is a classical approach for accelerating stochastic …
Stable Anderson Acceleration for Deep Learning
Anderson acceleration (AA) is an extrapolation technique designed to speed-up fixed-point iterations like those arising from the iterative training of DL models. Training DL models requires large datasets processed in ra…
Deep Learningimage-classificationImage ClassificationDamped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilization
Anderson mixing has been heuristically applied to reinforcement learning (RL) algorithms for accelerating convergence and improving the sampling efficiency of deep RL. Despite its heuristic improvement of convergence, a …
Deep Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Anderson Acceleration for Reinforcement Learning
Anderson acceleration is an old and simple method for accelerating the computation of a fixed point. However, as far as we know and quite surprisingly, it has never been applied to dynamic programming or reinforcement le…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)