Vision-Conditioned Variational Bayesian Last Layer Dynamics Models

2026-01-14 · Paul Brunzema, Thomas Lew, Ray Zhang, Takeru Shirasawa, John Subosits, Marcus Greiff arxiv

Agile control of robotic systems often requires anticipating how the environment affects system behavior. For example, a driver must perceive the road ahead to anticipate available friction and plan actions accordingly. Achieving such proactive adaptation within autonomous frameworks remains a challenge, particularly under rapidly changing conditions. Traditional modeling approaches often struggle to capture abrupt variations in system behavior, while adaptive methods are inherently reactive and may adapt too late to ensure safety. We propose a vision-conditioned variational Bayesian last-layer dynamics model that leverages visual context to anticipate changes in the environment. The model first learns nominal vehicle dynamics and is then fine-tuned with feature-wise affine transformations of latent features, enabling context-aware dynamics prediction. The resulting model is integrated into an optimal controller for vehicle racing. We validate our method on a Lexus LC500 racing through water puddles. With vision-conditioning, the system completed all 12 attempted laps under varying conditions. In contrast, all baselines without visual context consistently lost control, demonstrating the importance of proactive dynamics adaptation in high-performance applications.

📄 PDF Abstract BibTeX arXiv:2601.09178

Code (0)

등록된 구현이 없습니다.

Variational Bayesian Last Layers

2024-04-17 · James Harrison, John Willes, Jasper Snoek

We introduce a deterministic variational formulation for training Bayesian last layer neural networks. This yields a sampling-free, single-pass model and loss that effectively improves uncertainty estimation. Our variati…

Out-of-Distribution Detection Variational Inference

Flexible Bayesian Last Layer Models Using Implicit Priors and Diffusion Posterior Sampling

2024-08-07 · Jian Xu, Zhiqi Lin, Shigui Li, Min Chen 외

Bayesian Last Layer (BLL) models focus solely on uncertainty in the output layer of neural networks, demonstrating comparable performance to more complex Bayesian models. However, the use of Gaussian priors for last laye…

Computational Efficiency Out-of-Distribution Detection Uncertainty Quantification

Uncertainty-Aware Last-Layer Adaptation of RETFound for Referable Diabetic Retinopathy Screening Under Dataset Shift

2026-06-30 · Karim Mardhani arxiv

This paper presents a safety-centered empirical evaluation of uncertainty-aware last-layer adaptation for referable diabetic retinopathy screening using RETFound, a self-supervised vision-transformer retinal foundation m…

Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapters

2026-04-03 · Haotian Xiang, Bingcong Li, Qin Lu arxiv

When deploying large language models (LLMs) to safety-critical applications, uncertainty quantification (UQ) is of utmost importance to self-assess the reliability of the LLM-based decisions. However, such decisions typi…

parameter-efficient fine-tuning Bayesian Inference

Bayesian Optimization via Continual Variational Last Layer Training

2024-12-12 · Paul Brunzema, Mikkel Jordahn, John Willes, Sebastian Trimpe 외

Gaussian Processes (GPs) are widely seen as the state-of-the-art surrogate models for Bayesian optimization (BO) due to their ability to model uncertainty and their performance on tasks where correlations are easily capt…

Bayesian Optimization Gaussian Processes