Collective evolution of weights in wide neural networks
We derive a nonlinear integro-differential transport equation describing collective evolution of weights under gradient descent in large-width neural-network-like models. We characterize stationary points of the evolution and analyze several scenarios where the transport equation can be solved approximately. We test our general method in the special case of linear free-knot splines, and find good agreement between theory and experiment in observations of global optima, stability of stationary points, and convergence rates.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Evolution of Collective Decision-Making Mechanisms for Collective Perception
Autonomous robot swarms must be able to make fast and accurate collective decisions, but speed and accuracy are known to be conflicting goals. While collective decision-making is widely studied in swarm robotics research…
Decision MakingRoborobo! a Fast Robot Simulator for Swarm and Collective Robotics
Roborobo! is a multi-platform, highly portable, robot simulator for large-scale collective robotics experiments. Roborobo! is coded in C++, and follows the KISS guideline ("Keep it simple"). Therefore, its external depen…
A scaling law of multilevel evolution: how the balance between within- and among-collective evolution is determined
Numerous living systems are hierarchically organised, whereby replicating components are grouped into reproducing collectives -- e.g., organelles are grouped into cells, and cells are grouped into multicellular organisms…
Embodied Evolution in Collective Robotics: A Review
This paper provides an overview of evolutionary robotics techniques applied to on-line distributed evolution for robot collectives -- namely, embodied evolution. It provides a definition of embodied evolution as well as …
Deep Collective Learning: Learning Optimal Inputs and Weights Jointly in Deep Neural Networks
It is well observed that in deep learning and computer vision literature, visual data are always represented in a manually designed coding scheme (eg., RGB images are represented as integers ranging from 0 to 255 for eac…
image-classificationImage Classification