The boundary of neural network trainability is fractal
Some fractals -- for instance those associated with the Mandelbrot and quadratic Julia sets -- are computed by iterating a function, and identifying the boundary between hyperparameters for which the resulting series diverges or remains bounded. Neural network training similarly involves iterating an update function (e.g. repeated steps of gradient descent), can result in convergent or divergent behavior, and can be extremely sensitive to small changes in hyperparameters. Motivated by these similarities, we experimentally examine the boundary between neural network hyperparameters that lead to stable and divergent training. We find that this boundary is fractal over more than ten decades of scale in all tested configurations.
Code (1)
Similar Papers 제목 키워드 기반
Complex fractal trainability boundary can arise from trivial non-convexity
Training neural networks involves optimizing parameters to minimize a loss function, where the nature of the loss function and the optimization strategy are crucial for effective training. Hyperparameter choices, such as…
Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models
In the realm of fractal geometry, intricate structures emerge from simple iterative processes that partition parameter spaces into regions of stability and instability. Likewise, training large language models involves i…
DecoderA complete Leibniz-Mach cosmology: From the Leibniz Question to MOND
The claim that the large scale structure of the Universe is hierarchical has a very long history going back at least to Charlier's papers of the early 20th century. In recent years, the debate has centered largely on the…
Fractal and Regular Geometry of Deep Neural Networks
We study the geometric properties of random neural networks by investigating the boundary volumes of their excursion sets for different activation functions, as the depth increases. More specifically, we show that, for a…
Enhancing fractal descriptors on images by combining boundary and interior of Minkowski dilation
This work proposes to obtain novel fractal descriptors from gray-level texture images by combining information from interior and boundary measures of the Minkowski dilation applied to the texture surface. At first, the i…