홈 › Papers

Improving Sample Efficiency with Normalized RBF Kernels

2020-07-30 · Sebastian Pineda-Arango, David Obando-Paniagua, Alperen Dedeoglu, Philip Kurzendörfer, Friedemann Schestag, Randolf Scholz

In deep learning models, learning more with less data is becoming more important. This paper explores how neural networks with normalized Radial Basis Function (RBF) kernels can be trained to achieve better sample efficiency. Moreover, we show how this kind of output layer can find embedding spaces where the classes are compact and well-separated. In order to achieve this, we propose a two-phase method to train those type of neural networks on classification tasks. Experiments on CIFAR-10 and CIFAR-100 show that networks with normalized kernels as output layer can achieve higher sample efficiency, high compactness and well-separability through the presented method in comparison to networks with SoftMax output layer.

📄 PDF Abstract BibTeX arXiv:2007.15397

Code (1)

anonym-submission-2020/n-rbf-kernels 공식 구현 tf

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Kernelized Complete Conditional Stein Discrepancy

2019-04-09 · Raghav Singhal, Xintian Han, Saad Lahlou, Rajesh Ranganath

Much of machine learning relies on comparing distributions with discrepancy measures. Stein's method creates discrepancy measures between two distributions that require only the unnormalized density of one and samples fr…

On the Similarity between the Laplace and Neural Tangent Kernels

2020-07-03 · NeurIPS 2020 12 · Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun 외

Recent theoretical work has shown that massively overparameterized neural networks are equivalent to kernel regressors that use Neural Tangent Kernels(NTK). Experiments show that these kernel methods perform similarly to…

Spherical Random Features for Polynomial Kernels

2015-12-01 · NeurIPS 2015 12 · Jeffrey Pennington, Felix Xinnan X. Yu, Sanjiv Kumar

Compact explicit feature maps provide a practical framework to scale kernel methods to large-scale learning, but deriving such maps for many types of kernels remains a challenging open problem. Among the commonly used ke…

General Classification

Convex Deep Learning via Normalized Kernels

2014-12-01 · NeurIPS 2014 12 · Özlem Aslan, Xinhua Zhang, Dale Schuurmans

Deep learning has been a long standing pursuit in machine learning, which until recently was hampered by unreliable training methods before the discovery of improved heuristics for embedded layer training. A complementar…

Deep Learning

Mixed Variable Bayesian Optimization with Frequency Modulated Kernels

2021-02-25 · Changyong Oh, Efstratios Gavves, Max Welling

The sample efficiency of Bayesian optimization(BO) is often boosted by Gaussian Process(GP) surrogate models. However, on mixed variable spaces, surrogate models other than GPs are prevalent, mainly due to the lack of ke…

Bayesian Optimization Hyperparameter Optimization