Calibrating Transformers via Sparse Gaussian Processes

2023-03-04 · Wenlong Chen, Yingzhen Li

Transformer models have achieved profound success in prediction tasks in a wide range of applications in natural language processing, speech recognition and computer vision. Extending Transformer's success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer to calibrate its uncertainty. It replaces the scaled dot-product operation with a valid symmetric kernel and uses sparse Gaussian processes (SGP) techniques to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transformers achieve competitive predictive accuracy, while noticeably improving both in-distribution calibration and out-of-distribution robustness and detection.

📄 PDF Abstract BibTeX arXiv:2303.02444

Code (1)

chenw20/sgpa 공식 구현 pytorch

Tasks

Bayesian Inference Gaussian Processes speech-recognition Speech Recognitionvalid

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Gaussian Process Gaussian Processes are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for…

Calibrating Deep Convolutional Gaussian Processes

2018-05-26 · Gia-Lac Tran, Edwin V. Bonilla, John P. Cunningham, Pietro Michiardi 외

The wide adoption of Convolutional Neural Networks (CNNs) in applications where decision-making under uncertainty is fundamental, has brought a great deal of attention to the ability of these models to accurately quantif…

Decision Making Decision Making Under Uncertainty Gaussian Processes General Classification+2

Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes

2024-02-02 · Yingyi Chen, Qinghua Tao, Francesco Tonin, Johan A. K. Suykens

While the great capability of Transformers significantly boosts prediction accuracy, it could also yield overconfident predictions and require calibrated uncertainty estimation, which can be commonly tackled by Gaussian …

Gaussian Processes Variational Inference

Calibrating the Rigged Lottery: Making All Tickets Reliable

2023-02-18 · Bowen Lei, Ruqi Zhang, Dongkuan Xu, Bani Mallick

Although sparse training has been successfully used in various resource-limited deep learning tasks to save memory, accelerate training, and reduce inference time, the reliability of the produced sparse models remains un…

All Decision Making

Decoupled Sparse Gaussian Processes Components]{Decoupled Sparse Gaussian Processes Components : Separating Decision Making from Data Manifold Fitting

2020-11-23 · pproximateinference AABI Symposium 2021 1 · Sebastian Popescu, David J. Sharp, James H. Cole, Ben Glocker

We propose a decoupling in Reproducing Kernel Hilbert Space of the parametric and non-parametric components of Sparse Gaussian Processes. We demonstrate that this decoupling results in a set of inducing points concentra…

Decision Making Gaussian Processes regression

A Bayesian latent Gaussian process framework for aerodynamic uncertainty quantification

2026-06-27 · Geoffrey Davis, Ashwin Renganathan arxiv

Predicting the aerodynamic performance (e.g. lift, drag, and moment coefficients) of an aircraft is challenging -- computational models are biased and direct simulations are prohibitive. A pragmatic way to overcome this …