Learning a Decision Tree Algorithm with Transformers
Decision trees are renowned for their ability to achieve high predictive performance while remaining interpretable, especially on tabular data. Traditionally, they are constructed through recursive algorithms, where they partition the data at every node in a tree. However, identifying a good partition is challenging, as decision trees optimized for local segments may not yield global generalization. To address this, we introduce MetaTree, a transformer-based model trained via meta-learning to directly produce strong decision trees. Specifically, we fit both greedy decision trees and globally optimized decision trees on a large number of datasets, and train MetaTree to produce only the trees that achieve strong generalization performance. This training enables MetaTree to emulate these algorithms and intelligently adapt its strategy according to the context, thereby achieving superior generalization performance.
Code (1)
Tasks
Meta-LearningSimilar Papers 제목 키워드 기반
Generalizing Tree Models for Improving Prediction Accuracy
Can we generalize and improve the representation power of tree models? Tree models are often favored over deep neural networks due to their interpretable structures in problems where the interpretability is required, suc…
PredictionMedication Mention Detection in Tweets Using ELECTRA Transformers and Decision Trees
This study describes our proposed model design for the SMM4H 2020 Task 1. We fine-tune ELECTRA transformers using our trained SVM filter for data augmentation, along with decision trees to detect medication mentions in t…
Data AugmentationChallenging Gradient Boosted Decision Trees with Tabular Transformers for Fraud Detection at Booking.com
Transformer-based neural networks, empowered by Self-Supervised Learning (SSL), have demonstrated unprecedented performance across various domains. However, related literature suggests that tabular Transformers may strug…
Fraud DetectionSelection biasSelf-Supervised LearningTransformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes
Large language models (LLMs) perform remarkably well on tabular datasets in zero- and few-shot settings, since they can extract meaning from natural language column headers that describe features and labels. Similarly, T…
In-Context LearningNatural Language UnderstandingSALT : Sharing Attention between Linear layer and Transformer for tabular dataset
Handling tabular data with deep learning models is a challenging problem despite their remarkable success in vision and language processing applications. Therefore, many practitioners still rely on classical models such …
Deep Learning