Strengthening LLMs for Tabular Prediction with Structural Priors

2025-10-20 · Pengxiang Cai, Zihao Gao, Wanchen Lian, Guocong Li, Jintai Chen arxiv

Tabular prediction has long been dominated by gradient-boosted decision trees and specialized deep tabular models, while large language models (LLMs) remain difficult to make competitive despite their cross-task adaptability and transparent reasoning traces. We address this gap by incorporating tabular structural priors into LLM post-training. Specifically, we propose Permutation Relative Policy Optimization (PRPO), which operationalizes column-permutation invariance through label-preserving column permutations and two-level advantage estimation. This design converts sparse outcome rewards into denser and more stable optimization signals. Extensive experiments on 139 OpenML datasets show that our 8B model reaches a genuinely competitive regime against strong specialized tabular baselines. It achieves strong fully supervised performance, dominates zero-shot settings, and performs on par with 32-shot strong baselines. Moreover, it substantially outperforms much larger general-purpose and reasoning LLMs, including up to a 53.17% improvement over DeepSeek-R1 (685B). These results show that structural-prior RL post-training is an effective route for making LLMs competitive in tabular prediction.

📄 PDF Abstract BibTeX arXiv:2510.17385

Code (0)

등록된 구현이 없습니다.

Towards Understanding Layer Contributions in Tabular In-Context Learning Models

2025-11-19 · Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger arxiv

Despite the architectural similarities between tabular in-context learning (ICL) models and large language models (LLMs), little is known about how individual layers contribute to tabular prediction. In this paper, we in…

Model Compression

TAROT: Task-Adaptive Refinement of LLM-prior Graphs for Few-shot Tabular Learning

2026-06-10 · Ruxue Shi, Yili Wang, Mengnan Du, Hangting Ye 외 arxiv

Few-shot tabular learning provides a cost-effective approach for real-world applications where annotation is costly and collecting sufficient samples for new tasks is difficult. Existing Traditional and LLM-based methods…

Tabular LLMs for Interpretable Few-Shot Alzheimer's Disease Prediction with Multimodal Biomedical Data

2026-03-17 · Sophie Kearney, Shu Yang, Zixuan Wen, Weimin Lyu 외 arxiv

Accurate diagnosis of Alzheimer's disease (AD) requires handling tabular biomarker data, yet such data are often small and incomplete, where deep learning models frequently fail to outperform classical methods. Pretraine…

Incorporating LLM Priors into Tabular Learners

2023-11-20 · Max Zhu, Siniša Stanivuk, Andrija Petrovic, Mladen Nikolic 외

We present a method to integrate Large Language Models (LLMs) and traditional tabular data classification techniques, addressing LLMs challenges like data serialization sensitivity and biases. We introduce two strategies…

regression Sensitivity

The Illusion of Generalization in Tabular Language Models

2026-02-03 · Aditya Gorla, Ratish Puduppully arxiv

Tabular Language Models (TLMs) have been claimed to achieve strong generalization for tabular prediction. We conduct a systematic re-evaluation of Tabula-8B as a representative TLM, utilizing 165 datasets from the UniPre…