Towards a Relationship-Aware Transformer for Tabular Data
Deep learning models for tabular data typically do not allow for imposing a graph of external dependencies between samples, which can be useful for accounting for relatedness in tasks such as treatment effect estimation. Graph neural networks only consider adjacent nodes, making them difficult to apply to sparse graphs. This paper proposes several solutions based on a modified attention mechanism, which accounts for possible relationships between data points by adding a term to the attention matrix. Our models are compared with each other and the gradient boosting decision trees in a regression task on synthetic and real-world datasets, as well as in a treatment effect estimation task on the IHDP dataset.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Geometry-Aware Tabular Diffusion
Tabular synthesis is critical for privacy-preserving sharing and augmentation, yet diffusion models rely on implicit mechanisms to capture inter-column relationships. We introduce Geometry-Aware Tabular Diffusion (GATD),…
Deep Learning with Tabular Data: A Self-supervised Approach
We have described a novel approach for training tabular data using the TabTransformer model with self-supervised learning. Traditional machine learning models for tabular data, such as GBDT are being widely used though o…
Deep LearningSelf-Supervised LearningTabular data generation with tensor contraction layers and transformers
Generative modeling for tabular data has recently gained significant attention in the Deep Learning domain. Its objective is to estimate the underlying distribution of the data. However, estimating the underlying distrib…
Density EstimationTabular Data GenerationFATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
Sequential tabular data is one of the most commonly used data types in real-world applications. Different from conventional tabular data, where rows in a table are independent, sequential tabular data contains rich conte…
Language ModelingLanguage ModellingMasked Language ModelingExploring Transformer Placement in Variational Autoencoders for Tabular Data Generation
Tabular data remains a challenging domain for generative models. In particular, the standard Variational Autoencoder (VAE) architecture, typically composed of multilayer perceptrons, struggles to model relationships betw…
Tabular Data Generation