ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks

2018-11-26 · NeurIPS 2020 12 · Shuxuan Guo, Jose M. Alvarez, Mathieu Salzmann

We introduce an approach to training a given compact network. To this end, we leverage over-parameterization, which typically improves both neural network optimization and generalization. Specifically, we propose to expand each linear layer of the compact network into multiple consecutive linear layers, without adding any nonlinearity. As such, the resulting expanded network, or ExpandNet, can be contracted back to the compact one algebraically at inference. In particular, we introduce two convolutional expansion strategies and demonstrate their benefits on several tasks, including image classification, object detection, and semantic segmentation. As evidenced by our experiments, our approach outperforms both training the compact network from scratch and performing knowledge distillation from a teacher. Furthermore, our linear over-parameterization empirically reduces gradient confusion during training and improves the network generalization.

📄 PDF Abstract BibTeX arXiv:1811.10495

Code (0)

등록된 구현이 없습니다.

Tasks

General Classification image-classification Image Classification Knowledge Distillation object-detection Object Detection Semantic Segmentation Transfer Learning

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Spectral Tensor Train Parameterization of Deep Learning Layers

2021-03-07 · Anton Obukhov, Maxim Rakhuba, Alexander Liniger, Zhiwu Huang 외

We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when c…

Deep Learning image-classification Image Classification Image Generation+1

The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

2020-02-07 · ICML 2020 1 · Jakub Swiatkowski, Kevin Roth, Bastiaan S. Veeling, Linh Tran 외

Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights. Recent work developing this class of methods has explored ever richer parameterizati…

Bayesian Inference Variational Inference

Deep Generative Model for Efficient 3D Airfoil Parameterization and Generation

2021-01-07 · Wei Chen, Arun Ramamurthy

In aerodynamic shape optimization, the convergence and computational cost are greatly affected by the representation capacity and compactness of the design space. Previous research has demonstrated that using a deep gene…

On the Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

2019-09-25 · Jakub Świątkowski, Kevin Roth, Bastiaan S. Veeling, Linh Tran 외

Variational Bayesian Inference is a popular methodology for approximating posterior distributions in Bayesian neural networks. Recent work developing this class of methods has explored ever richer parameterizations of th…