Statistical learning theory and Occam's razor: The core argument
Statistical learning theory is often associated with the principle of Occam's razor, which recommends a simplicity preference in inductive inference. This paper distills the core argument for simplicity obtainable from statistical learning theory, built on the theory's central learning guarantee for the method of empirical risk minimization. This core "means-ends" argument is that a simpler hypothesis class or inductive model is better because it has better learning guarantees; however, these guarantees are model-relative and so the theoretical push towards simplicity is checked by our prior knowledge.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning TheorySimilar Papers 제목 키워드 기반
Likelihood-free Model Choice for Simulator-based Models with the Jensen--Shannon Divergence
Choice of appropriate structure and parametric dimension of a model in the light of data has a rich history in statistical research, where the first seminal approaches were developed in 1970s, such as the Akaike's and Sc…
Benign interpolation and Occam's razor
Contemporary deep learning methods generalize well even when they fit their training data perfectly, a phenomenon known as benign interpolation. This phenomenon cannot be accounted for by classical statistical learning t…
In-context learning and Occam's razor
A central goal of machine learning is generalization. While the No Free Lunch Theorem states that we cannot obtain theoretical guarantees for generalization without further assumptions, in practice we observe that simple…
Data CompressionIn-Context LearningThe Geometric Occam's Razor Implicit in Deep Learning
In over-parameterized deep neural networks there can be many possible parameter configurations that fit the training data exactly. However, the properties of these interpolating solutions are poorly understood. We argue …
ARCDeep LearningGrounding Occam's Razor in a Formal Theory of Simplicity
A formal theory of simplicity is introduced, in the context of a "combinational" computation model that views computation as comprising the iterated transformational and compositional activity of a population of agents u…