Deep Neural Nets with Interpolating Function as Output Activation

2018-02-01 · NeurIPS 2018 12 · Bao Wang, Xiyang Luo, Zhen Li, Wei Zhu, Zuoqiang Shi, Stanley J. Osher

We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function. And we propose end-to-end training and testing algorithms for this new architecture. Compared to classical neural nets with softmax function as output activation, the surrogate with interpolating function as output activation combines advantages of both deep and manifold learning. The new framework demonstrates the following major advantages: First, it is better applicable to the case with insufficient training data. Second, it significantly improves the generalization accuracy on a wide variety of networks. The algorithm is implemented in PyTorch, and code will be made publicly available.

📄 PDF Abstract BibTeX arXiv:1802.00168

Code (1)

BaoWangMath/DNN-DataDependentActivation pytorch

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning

2019-07-16 · Bao Wang, Stanley J. Osher

Improving the accuracy and robustness of deep neural nets (DNNs) and adapting them to small training data are primary tasks in deep learning research. In this paper, we replace the output activation function of DNNs, typ…

Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization

2018-09-23 · Bao Wang, Alex T. Lin, Wei Zhu, Penghang Yin 외

We improve the robustness of Deep Neural Net (DNN) to adversarial attacks by using an interpolating function as the output activation. This data-dependent activation remarkably improves both the generalization and robust…

Adversarial Attack Adversarial Defense Data Augmentation

Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

2018-09-24 · Wojciech Tarnowski, Piotr Warchoł, Stanisław Jastrzębski, Jacek Tabor 외

We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theor…

P-nets: Deep Polynomial Neural Networks

2020-06-01 · CVPR 2020 6 · Grigorios G. Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis 외

Deep Convolutional Neural Networks (DCNNs) is currently the method of choice both for generative, as well as for discriminative learning in computer vision and machine learning. The success of DCNNs can be attributed to …

Image Generation Polynomial Neural Networks

$Π-$nets: Deep Polynomial Neural Networks

2020-03-08 · Grigorios G. Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis 외

Audio Classification Graph Representation Learning Image Classification Image Generation+1