Unified Probabilistic Neural Architecture and Weight Ensembling Improves Model Robustness

2022-10-08 · Sumegha Premchandar, Sandeep Madireddy, Sanket Jantre, Prasanna Balaprakash

Robust machine learning models with accurately calibrated uncertainties are crucial for safety-critical applications. Probabilistic machine learning and especially the Bayesian formalism provide a systematic framework to incorporate robustness through the distributional estimates and reason about uncertainty. Recent works have shown that approximate inference approaches that take the weight space uncertainty of neural networks to generate ensemble prediction are the state-of-the-art. However, architecture choices have mostly been ad hoc, which essentially ignores the epistemic uncertainty from the architecture space. To this end, we propose a Unified probabilistic architecture and weight ensembling Neural Architecture Search (UraeNAS) that leverages advances in probabilistic neural architecture search and approximate Bayesian inference to generate ensembles form the joint distribution of neural network architectures and weights. The proposed approach showed a significant improvement both with in-distribution (0.86% in accuracy, 42% in ECE) CIFAR-10 and out-of-distribution (2.43% in accuracy, 30% in ECE) CIFAR-10-C compared to the baseline deterministic approach.

📄 PDF Abstract BibTeX arXiv:2210.04083

Code (0)

등록된 구현이 없습니다.

Tasks

Bayesian Inference Neural Architecture Search

Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

2017-03-06 · NeurIPS 2017 12 · Antti Tarvainen, Harri Valpola

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, an…

Semi-Supervised Image Classification Semi-Supervised RGBD Semantic Segmentation Semi-Supervised Semantic Segmentation Source Free Object Detection

Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification

2021-04-27 · Yixiao Ge, Xiao Zhang, Ching Lam Choi, Ka Chun Cheung 외

The recent studies of knowledge distillation have discovered that ensembling the "dark knowledge" from multiple teachers or students contributes to creating better soft targets for training, but at the cost of significan…

Classification General Classification Knowledge Distillation

Multimodal Object Detection via Probabilistic Ensembling

2021-04-07 · Yi-Ting Chen, Jinghao Shi, Zelin Ye, Christoph Mertz 외

Object detection with multimodal inputs can improve many safety-critical systems such as autonomous vehicles (AVs). Motivated by AVs that operate in both day and night, we study multimodal object detection with RGB and t…

3D Object Detection Autonomous Vehicles Object object-detection+2

The Professor: Multi-Teacher Unsupervised Prompt Distillation for Vision-Language Models

2026-06-22 · Ahmad Algadhi, Ahmed Alzuhair, Omar Alkhulaif, Muzammil Behzad arxiv

Prompt distillation compresses large vision-language models (VLMs) such as CLIP into lightweight student models by matching teacher predictions on unlabeled domain images. PromptKD (CVPR 2024) established this paradigm w…

Effective training-time stacking for ensembling of deep neural networks

2022-06-27 · Polina Proscura, Alexey Zaytsev

Ensembling is a popular and effective method for improving machine learning (ML) models. It proves its value not only in classical ML but also for deep learning. Ensembles enhance the quality and trustworthiness of ML so…

Deep Learning