홈 › Papers

Retrieval & Fine-Tuning for In-Context Tabular Models

2024-06-07 · Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, Anthony Caterini

Tabular data is a pervasive modality spanning a wide range of domains, and the inherent diversity poses a considerable challenge for deep learning. Recent advancements using transformer-based in-context learning have shown promise on smaller and less complex datasets, but have struggled to scale to larger and more complex ones. To address this limitation, we propose a combination of retrieval and fine-tuning: we can adapt the transformer to a local subset of the data by collecting nearest neighbours, and then perform task-specific fine-tuning with this retrieved set of neighbours in context. Using TabPFN as the base model -- currently the best tabular in-context learner -- and applying our retrieval and fine-tuning scheme on top results in what we call a locally-calibrated PFN, or LoCalPFN. We conduct extensive evaluation on 95 datasets curated by TabZilla from OpenML, upon which we establish a new state-of-the-art with LoCalPFN -- even with respect to tuned tree-based models. Notably, we show a significant boost in performance compared to the base in-context model, demonstrating the efficacy of our approach and advancing the frontier of deep learning in tabular data.

📄 PDF Abstract BibTeX arXiv:2406.05207

Code (0)

등록된 구현이 없습니다.

Tasks

DiversityIn-Context LearningRetrieval

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

BASE 설명 없음

TABPFN We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive…

On Finetuning Tabular Foundation Models

2025-06-10 · Ivan Rubachev, Akim Kotelnikov, Nikolay Kartashev, Artem Babenko

Foundation models are an emerging research direction in tabular deep learning. Notably, TabPFNv2 recently claimed superior performance over traditional GBDT-based methods on small-scale datasets using an in-context learn…

In-Context Learning Retrieval

Enhancing Knowledge Retrieval with In-Context Learning and Semantic Search through Generative AI

2024-06-13 · Mohammed-Khalil Ghali, Abdelrahman Farrag, Daehan Won, Yu Jin

Retrieving and extracting knowledge from extensive research documents and large databases presents significant challenges for researchers, students, and professionals in today's information-rich era. Existing retrieval s…

In-Context Learning Information Retrieval Retrieval Text Retrieval

Fine-Tuning the Retrieval Mechanism for Tabular Deep Learning

2023-11-13 · Felix den Breejen, Sangmin Bae, Stephen Cha, Tae-Young Kim 외

While interests in tabular deep learning has significantly grown, conventional tree-based models still outperform deep learning methods. To narrow this performance gap, we explore the innovative retrieval mechanism, a me…

Deep Learning Retrieval Transfer Learning

ARASH: Adaptive Retrieval And Shot Selection for Tabular Prediction

2026-08-18 · Samirasadat Jamalidinan, Yue Xu, Kazem Cheshmi arxiv

Tabular prediction is a critical task across numerous applications. The recent success of large language models has sparked various approaches for adapting them to the tabular domain. A prevalent strategy involves traini…

Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models

2025-02-05 · Xumeng Wen, Shun Zheng, Zhen Xu, Yiming Sun 외

Recent studies have shown that large language models (LLMs), when customized with post-training on tabular data, can acquire general tabular in-context learning (TabICL) capabilities. These models are able to transfer ef…