There Will Be a Scientific Theory of Deep Learning
In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Minds, Brains, AI
In the last year or so and going back many decades there has been extensive claims by major computational scientists, engineers, and others that AGI, artificial general intelligence, is five or ten years away, but withou…
Self-Driving CarsAn introduction to computational complexity and statistical learning theory applied to nuclear models
The fact that we can build models from data, and therefore refine our models with more data from experiments, is usually given for granted in scientific inquiry. However, how much information can we extract, and how prec…
Learning TheoryPrototype matching: children's preference for forming scientific concepts
Inspired by a sample lesson, this paper studies and discusses children's preferences in learning scientific concepts. In a "Dissolution" lesson, one of the students took the demonstration experiment of "carmine dissolves…
Data-Adaptive Dimensional Analysis for Accurate Interpolation and Extrapolation in Computer Experiments
Dimensional analysis (DA) pays attention to fundamental physical dimensions such as length and mass when modelling scientific and engineering systems. It goes back at least a century to Buckingham's Pi theorem, which cha…
validUnified Growth Theory Contradicted by the Economic Growth in Latin America
Historical economic growth in Latin America is analysed using the data of Maddison. Unified Growth Theory is found to be contradicted by these data in the same way as it is contradicted by the economic growth in Africa, …