To Predict or Not To Predict? Proportionally Masked Autoencoders for Tabular Data Imputation

2024-12-26 · Jungkyu Kim, Kibok Lee, Taeyoung Park

Masked autoencoders (MAEs) have recently demonstrated effectiveness in tabular data imputation. However, due to the inherent heterogeneity of tabular data, the uniform random masking strategy commonly used in MAEs can disrupt the distribution of missingness, leading to suboptimal performance. To address this, we propose a proportional masking strategy for MAEs. Specifically, we first compute the statistics of missingness based on the observed proportions in the dataset, and then generate masks that align with these statistics, ensuring that the distribution of missingness is preserved after masking. Furthermore, we argue that simple MLP-based token mixing offers competitive or often superior performance compared to attention mechanisms while being more computationally efficient, especially in the tabular domain with the inherent heterogeneity. Experimental results validate the effectiveness of the proposed proportional masking strategy across various missing data patterns in tabular datasets. Code is available at: \url{https://github.com/normal-kim/PMAE}.

📄 PDF Abstract BibTeX arXiv:2412.19152

Code (1)

normal-kim/pmae 공식 구현 pytorch

Tasks

Imputation

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

ReMasker: Imputing Tabular Data with Masked Autoencoding

2023-09-25 · Tianyu Du, Luca Melis, Ting Wang

We present ReMasker, a new method of imputing missing values in tabular data by extending the masked autoencoding framework. Compared with prior work, ReMasker is both simple -- besides the missing values (i.e., naturall…

Imputation Missing Values

Bootstrapped Masked Autoencoders for Vision BERT Pretraining

2022-07-14 · Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen 외

We propose bootstrapped masked autoencoders (BootMAE), a new approach for vision BERT pretraining. BootMAE improves the original masked autoencoders (MAE) with two core designs: 1) momentum encoder that provides online f…

Decoder Object Detection Prediction Self-Supervised Image Classification+1

A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond

2022-07-30 · Chaoning Zhang, Chenshuang Zhang, Junha Song, John Seon Keun Yi 외

Masked autoencoders are scalable vision learners, as the title of MAE \cite{he2022masked}, which suggests that self-supervised learning (SSL) in vision might undertake a similar trajectory as in NLP. Specifically, genera…

Contrastive Learning Denoising Self-Supervised Learning

PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders

2024-08-16 · Xiangdong Zhang, Shaofeng Zhang, Junchi Yan

Masked autoencoder has been widely explored in point cloud self-supervised learning, whereby the point cloud is generally divided into visible and masked parts. These methods typically include an encoder accepting visibl…

3D Object Classification 3D Point Cloud Classification Decoder Few-Shot 3D Point Cloud Classification+4

3D Cloud reconstruction through geospatially-aware Masked Autoencoders

2025-01-03 · Stella Girtsou, Emiliano Diaz Salas-Porras, Lilli Freischem, Joppe Massant 외

Clouds play a key role in Earth's radiation balance with complex effects that introduce large uncertainties into climate models. Real-time 3D cloud data is essential for improving climate predictions. This study leverage…

Self-Supervised Learning