Why can genetic algorithms work in high-dimensional search spaces?
We show that the effective dynamics of the elitist $(1+M)$ genetic algorithm is, in the limit of small mutations, clipped gradient descent on the loss in the presence of anisotropic Gaussian white noise. In expectation, therefore, a simple mutation-selection genetic algorithm follows the gradient of the loss, without explicit calculation of gradients and without averaging over loss evaluations. The genetic algorithm is slower than gradient descent because of the noise that acts in directions transverse to the gradient. However, this slowdown is controlled not by the number of parameters of the search space but by the effective rank of the Hessian of the loss function. For the concentrated Hessian spectra observed in neural-network loss functions the effective rank can be far smaller than the number of parameters, which may explain why genetic algorithms can scale to large search spaces.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
HYB-PARSIMONY: A hybrid approach combining Particle Swarm Optimization and Genetic Algorithms to find parsimonious models in high-dimensional datasets
The PSO-PARSIMONY methodology (a heuristic for finding accurate and low-complexity models with particle swarm optimization (PSO)) allows obtaining machine learning models with a good balance between accuracy and complexi…
ELENA: Epigenetic Learning through Evolved Neural Adaptation
Despite the success of metaheuristic algorithms in solving complex network optimization problems, they often struggle with adaptation, especially in dynamic or high-dimensional search spaces. Traditional approaches can b…
Traveling Salesman ProblemGenetically Modified Wolf Optimization with Stochastic Gradient Descent for Optimising Deep Neural Networks
When training Convolutional Neural Networks (CNNs) there is a large emphasis on creating efficient optimization algorithms and highly accurate networks. The state-of-the-art method of optimizing the networks is done by u…
An Evolutional Neural Network Framework for Classification of Microarray Data
DNA microarray gene-expression data has been widely used to identify cancerous gene signatures. Microarray can increase the accuracy of cancer diagnosis and prognosis. However, analyzing the large amount of gene expressi…
feature selectionPrognosisCan Genetic Programming Do Manifold Learning Too?
Exploratory data analysis is a fundamental aspect of knowledge discovery that aims to find the main characteristics of a dataset. Dimensionality reduction, such as manifold learning, is often used to reduce the number of…
Dimensionality Reduction