홈 › Papers

Tabular Transformers for Modeling Multivariate Time Series

2020-11-03 · Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, Erik Altman

Tabular datasets are ubiquitous in data science applications. Given their importance, it seems natural to apply state-of-the-art deep learning algorithms in order to fully unlock their potential. Here we propose neural network models that represent tabular time series that can optionally leverage their hierarchical structure. This results in two architectures for tabular time series: one for learning representations that is analogous to BERT and can be pre-trained end-to-end and used in downstream tasks, and one that is akin to GPT and can be used for generation of realistic synthetic tabular sequences. We demonstrate our models on two datasets: a synthetic credit card transaction dataset, where the learned representations are used for fraud detection and synthetic data generation, and on a real pollution dataset, where the learned encodings are used to predict atmospheric pollutant concentrations. Code and data are available at https://github.com/IBM/TabFormer.

📄 PDF Abstract BibTeX arXiv:2011.01843

Code (1)

IBM/TabFormer 공식 구현 pytorch

Tasks

Fraud Detection Synthetic Data GenerationTime SeriesTime Series Analysis

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

Linear Warmup With Cosine Annealing Linear Warmup With Cosine Annealing is a learning rate schedule where we increase the learning rate linearly for $n$ updates and then anneal according to a cosine schedule…

Discriminative Fine-Tuning Discriminative Fine-Tuning is a fine-tuning strategy that is used for ULMFiT type models. Instead of using the same learning rate…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

WordPiece 설명 없음

Zero-shot Multivariate Time Series Forecasting Using Tabular Prior Fitted Networks

2026-04-09 · Mayuka Jayawardhana, Nihal Sharma, Kazem Meidani, Bayan Bruss 외 arxiv

Tabular foundation models, particularly Prior-data Fitted Networks like TabPFN have emerged as the leading contender in a myriad of tasks ranging from data imputation to label prediction on the tabular data format surpas…

Multivariate Time Series Forecasting Univariate Time Series Forecasting

Attention Augmented Convolutional Transformer for Tabular Time-series

2021-10-05 · Sharath M Shankaranarayana, Davor Runje

Time-series classification is one of the most frequently performed tasks in industrial data science, and one of the most widely used data representation in the industrial setting is tabular representation. In this work, …

Language Modeling Language Modelling Quantization Representation Learning+3

Diffusion Transformers for Tabular Data Time Series Generation

2025-04-10 · Fabrizio Garuti, Enver Sangineto, Simone Luetto, Lorenzo Forni 외

Tabular data generation has recently attracted a growing interest due to its different application scenarios. However, generating time series of tabular data, where each element of the series depends on the others, remai…

Tabular Data Generation Time Series Time Series Generation Video Generation

Forging Time Series with Language: A Large Language Model Approach to Synthetic Data Generation

2025-05-21 · Cécile Rousseau, Tobia Boschi, Giandomenico Cornacchia, Dhaval Salwala 외

SDForger is a flexible and efficient framework for generating high-quality multivariate time series using LLMs. Leveraging a compact data representation, SDForger provides synthetic time series generation from a few samp…

Language Modeling Language Modelling Large Language Model Synthetic Data Generation+2

Deep Generative Transformers for Probabilistic Time Series and Spatiotemporal Forecasting

2026-03-12 · Rajdeep Pathak, Rahul Goswami, Madhurima Panja, Palash Ghosh 외 arxiv

Reliable uncertainty quantification is paramount for forecasting multivariate time series and spatiotemporal data. While Transformer architectures excel at sequence modeling, current probabilistic approaches typically re…