Towards Understanding Layer Contributions in Tabular In-Context Learning Models

2025-11-19 · Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger arxiv

Despite the architectural similarities between tabular in-context learning (ICL) models and large language models (LLMs), little is known about how individual layers contribute to tabular prediction. In this paper, we investigate how the latent spaces evolve across layers in tabular ICL models, identify potential redundant layers, and compare these dynamics with those observed in LLMs. We analyze TabPFN and TabICL through the "layers as painters" perspective, finding that only subsets of layers share a common representational language, suggesting structural redundancy and offering opportunities for model compression and improved interpretability.

📄 PDF Abstract BibTeX arXiv:2511.15432

Code (0)

등록된 구현이 없습니다.

Tasks

Model Compression

A Closer Look into LLMs for Table Understanding

2026-03-16 · Jia Wang, Chuanyu Qin, Mingyu Zheng, Qingyi Si 외 arxiv

Despite the success of Large Language Models (LLMs) in table understanding, their internal mechanisms remain unclear. In this paper, we conduct an empirical study on 16 LLMs, covering general LLMs, specialist tabular LLM…

Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

2026-05-07 · Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger arxiv

Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study…

Hardness in Markov Decision Processes: Theory and Practice

2022-10-24 · Michelangelo Conserva, Paulo Rauber

Meticulously analysing the empirical strengths and weaknesses of reinforcement learning methods in hard (challenging) environments is essential to inspire innovations and assess progress in the field. In tabular reinforc…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Multi-Layer Attention-Based Explainability via Transformers for Tabular Data

2023-02-28 · Andrea Treviño Gavito, Diego Klabjan, Jean Utke

We propose a graph-oriented attention-based explainability method for tabular data. Tasks involving tabular data have been solved mostly using traditional tree-based machine learning models which have the challenges of f…

feature selection

ConTextTab: A Semantics-Aware Tabular In-Context Learner

2025-06-12 · Marco Spinaci, Marek Polewczyk, Maximilian Schambach, Sam Thelin

Tabular in-context learning (ICL) has recently achieved state-of-the-art (SOTA) performance on several tabular prediction tasks. Previously restricted to classification problems on small tables, recent advances such as T…

In-Context Learning World Knowledge