On Compression Principle and Bayesian Optimization for Neural Networks
Finding methods for making generalizable predictions is a fundamental problem of machine learning. By looking into similarities between the prediction problem for unknown data and the lossless compression we have found an approach that gives a solution. In this paper we propose a compression principle that states that an optimal predictive model is the one that minimizes a total compressed message length of all data and model definition while guarantees decodability. Following the compression principle we use Bayesian approach to build probabilistic models of data and network definitions. A method to approximate Bayesian integrals using a sequence of variational approximations is implemented as an optimizer for hyper-parameters: Bayesian Stochastic Gradient Descent (BSGD). Training with BSGD is completely defined by setting only three parameters: number of epochs, the size of the dataset and the size of the minibatch, which define a learning rate and a number of iterations. We show that dropout can be used for a continuous dimensionality reduction that allows to find optimal network dimensions as required by the compression principle.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian OptimizationDimensionality ReductionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Data Compression with Bayesian Attention Networks
The lossless data compression algorithm based on Bayesian Attention Networks is derived from first principles.
Data CompressionConstraint-Aware Deep Neural Network Compression
Deep neural network compression has the potential to bring modern resource-hungry deep networks to resource-limited devices. However, in many of the most compelling deployment scenarios of compressed deep networks, the o…
Bayesian OptimizationNeural Network CompressionPedestrian DetectionBayesian Compression for Deep Learning
Compression and computational efficiency in deep learning have become a problem of great significance. In this work, we argue that the most principled and effective way to attack this problem is by adopting a Bayesian po…
Computational EfficiencyDeep LearningDirected Graph Grammars for Sequence-based Learning
Directed acyclic graphs (DAGs) are a class of graphs commonly used in practice, with examples that include electronic circuits, Bayesian networks, and neural architectures. While many effective encoders exist for DAGs, i…
Bayesian OptimizationGraph GenerationProperty PredictionA Bayesian Optimization Framework for Neural Network Compression
Neural network compression is an important step for deploying neural networks where speed is of high importance, or on devices with limited memory. It is necessary to tune compression parameters in order to achieve the d…
Bayesian OptimizationKnowledge DistillationNeural Network Compression