PFNs4BO: In-Context Learning for Bayesian Optimization

2023-05-27 · Samuel Müller, Matthias Feurer, Noah Hollmann, Frank Hutter

In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO). PFNs are neural processes that are trained to approximate the posterior predictive distribution (PPD) through in-context learning on any prior distribution that can be efficiently sampled from. We describe how this flexibility can be exploited for surrogate modeling in BO. We use PFNs to mimic a naive Gaussian process (GP), an advanced GP, and a Bayesian Neural Network (BNN). In addition, we show how to incorporate further information into the prior, such as allowing hints about the position of optima (user priors), ignoring irrelevant dimensions, and performing non-myopic BO by learning the acquisition function. The flexibility underlying these extensions opens up vast possibilities for using PFNs for BO. We demonstrate the usefulness of PFNs for BO in a large-scale evaluation on artificial GP samples and three different hyperparameter optimization testbeds: HPO-B, Bayesmark, and PD1. We publish code alongside trained models at github.com/automl/PFNs4BO.

📄 PDF Abstract BibTeX arXiv:2305.17535

Code (1)

automl/pfns4bo 공식 구현 pytorch

Tasks

AutoMLBayesian OptimizationHyperparameter OptimizationIn-Context Learning

Methods 이 논문이 사용한 방법론

Gaussian Process Gaussian Processes are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for…

Transformers Can Do Bayesian Inference

2021-12-20 · ICLR 2022 4 · Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka 외

Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks …

AutoML Bayesian Inference Few-Shot Image Classification Gaussian Processes+3

Frequentist Consistency of Prior-Data Fitted Networks for Causal Inference

2026-03-12 · Valentyn Melnychuk, Vahid Balazadeh, Stefan Feuerriegel, Rahul G. Krishnan arxiv

Foundation models based on prior-data fitted networks (PFNs) have shown strong empirical performance in causal inference by framing the task as an in-context learning problem. However, it is unclear whether PFN-based cau…

Causal Inference

TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks

2024-02-17 · Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde 외

While tabular classification has traditionally relied on from-scratch training, a recent breakthrough called prior-data fitted networks (PFNs) challenges this approach. Similar to large language models, PFNs make use of …

Fairness In-Context Learning parameter-efficient fine-tuning tabular-classification

Position: The Future of Bayesian Prediction Is Prior-Fitted

2025-05-29 · Samuel Müller, Arik Reuter, Noah Hollmann, David Rügamer 외

Training neural networks on randomly generated artificial datasets yields Bayesian models that capture the prior defined by the dataset-generating distribution. Prior-data Fitted Networks (PFNs) are a class of methods de…

Bayesian Inference Position

General Proximal Flow Networks

2026-02-28 · Alexander Strunk, Roland Assam arxiv

This paper introduces General Proximal Flow Networks (GPFNs), a generalization of Bayesian Flow Networks that broadens the class of admissible belief-update operators. In Bayesian Flow Networks, each update step is a Bay…