AdaSwarm: Augmenting Gradient-Based optimizers in Deep Learning with Swarm Intelligence
This paper introduces AdaSwarm, a novel gradient-free optimizer which has similar or even better performance than the Adam optimizer adopted in neural networks. In order to support our proposed AdaSwarm, a novel Exponentially weighted Momentum Particle Swarm Optimizer (EMPSO), is proposed. The ability of AdaSwarm to tackle optimization problems is attributed to its capability to perform good gradient approximations. We show that, the gradient of any function, differentiable or not, can be approximated by using the parameters of EMPSO. This is a novel technique to simulate GD which lies at the boundary between numerical methods and swarm intelligence. Mathematical proofs of the gradient approximation produced are also provided. AdaSwarm competes closely with several state-of-the-art (SOTA) optimizers. We also show that AdaSwarm is able to handle a variety of loss functions during backpropagation, including the maximum absolute error (MAE).
Code (2)
Tasks
Mathematical ProofsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Modified swarm-based metaheuristics enhance Gradient Descent initialization performance: Application for EEG spatial filtering
Gradient Descent (GD) approximators often fail in the solution space with multiple scales of convexities, i.e., in subspace learning and neural network scenarios. To handle that, one solution is to run GD multiple times …
EEGElectroencephalogram (EEG)Color Image Segmentation Using Multi-Objective Swarm Optimizer and Multi-level Histogram Thresholding
Rapid developments in swarm intelligence optimizers and computer processing abilities make opportunities to design more accurate, stable, and comprehensive methods for color image segmentation. This paper presents a new …
Image SegmentationSegmentationSemantic SegmentationUnsupervised Image SegmentationA Swarm Variant for the Schrödinger Solver
This paper introduces application of the Exponentially Averaged Momentum Particle Swarm Optimization (EM-PSO) as a derivative-free optimizer for Neural Networks. It adopts PSO's major advantages such as search space expl…
Mathematical ProofsLarge Language Models as Particle Swarm Optimizers
Optimization problems often require domain-specific expertise to design problem-dependent methodologies. Recently, several approaches have gained attention by integrating large language models (LLMs) into genetic algorit…
Symbolic RegressionTraveling Salesman ProblemHamiltonian Monte Carlo Particle Swarm Optimizer
We introduce the Hamiltonian Monte Carlo Particle Swarm Optimizer (HMC-PSO), an optimization algorithm that reaps the benefits of both Exponentially Averaged Momentum PSO and HMC sampling. The coupling of the position an…
Position