Know Where To Drop Your Weights: Towards Faster Uncertainty Estimation

2020-10-27 · NeurIPS Workshop ICBINB 2020 12 · Akshatha Kamath, Dwaraknath Gnaneshwar, Matias Valdenegro-Toro

Estimating epistemic uncertainty of models used in low-latency applications and Out-Of-Distribution samples detection is a challenge due to the computationally demanding nature of uncertainty estimation techniques. Estimating model uncertainty using approximation techniques like Monte Carlo Dropout (MCD), DropConnect (MCDC) requires a large number of forward passes through the network, rendering them inapt for low-latency applications. We propose Select-DC which uses a subset of layers in a neural network to model epistemic uncertainty with MCDC. Through our experiments, we show a significant reduction in the GFLOPS required to model uncertainty, compared to Monte Carlo DropConnect, with marginal trade-off in performance. We perform a suite of experiments on CIFAR 10, CIFAR 100, and SVHN datasets with ResNet and VGG models. We further show how applying DropConnect to various layers in the network with different drop probabilities affects the networks performance and the entropy of the predictive distribution.

📄 PDF Abstract BibTeX arXiv:2010.14019

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Monte Carlo Dropout 설명 없음

Average Pooling 설명 없음

Residual Connection 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Kaiming Initialization 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

SWA Object Detection

2020-12-23 · Haoyang Zhang, Ying Wang, Feras Dayoub, Niko Sünderhauf

Do you want to improve 1.0 AP for your object detector without any inference cost and any change to your detector? Let us tell you such a recipe. It is surprisingly simple: train your detector for an extra 12 epochs usin…

Instance Segmentation Object object-detection Object Detection+1

Token Merging: Your ViT But Faster

2022-10-17 · Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang 외

We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matc…

Efficient ViTs

Learning to Bid Without Knowing your Value

2017-11-03 · Zhe Feng, Chara Podimata, Vasilis Syrgkanis

We address online learning in complex auction settings, such as sponsored search auctions, where the value of the bidder is unknown to her, evolving in an arbitrary manner and observed only if the bidder wins an allocati…

ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages

2023-06-02 · Andrew Jesson, Chris Lu, Gunshi Gupta, Nicolas Beltran-Velez 외

This paper proposes a step toward approximate Bayesian inference in on-policy actor-critic deep reinforcement learning. It is implemented through three changes to the Asynchronous Advantage Actor-Critic (A3C) algorithm: …

Bayesian Inference continuous-control Continuous Control Deep Reinforcement Learning+4

Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors

2022-05-20 · Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor 외

Deep learning is increasingly moving towards a transfer learning paradigm whereby large foundation models are fine-tuned on downstream tasks, starting from an initialization learned on the source task. But an initializat…

Deep Learning Transfer Learning