Tricks and Plug-ins for Gradient Boosting with Transformers
Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augments transformers with boosting principles through subgrid token selection and importance-weighted sampling. Our method incorporates a least square boosting objective directly into the transformer pipeline, enabling more efficient training and improved performance. Across multiple fine-grained text classification benchmarks, BoostTransformer demonstrates both faster convergence and higher accuracy, surpassing standard transformers while minimizing architectural search overhead.
Code (0)
등록된 구현이 없습니다.
Tasks
Text ClassificationSimilar Papers 제목 키워드 기반
Tricks and Plugins to GBM on Images and Sequences
Convolutional neural networks (CNNs) and transformers, which are composed of multiple processing layers and blocks to learn the representations of data with multiple abstract levels, are the most successful machine learn…
feature selectionTricks and Plug-ins for Gradient Boosting in Image Classification
Convolutional Neural Networks (CNNs) have achieved remarkable success across a wide range of machine learning tasks by leveraging hierarchical feature learning through deep architectures. However, the large number of lay…
Image ClassificationGradient Boosting With Piece-Wise Linear Regression Trees
Gradient Boosted Decision Trees (GBDT) is a very successful ensemble learning algorithm widely used across a variety of applications. Recently, several variants of GBDT training algorithms and implementations have been d…
Ensemble LearningregressionTransformer tricks: Precomputing the first layer
This micro-paper describes a trick to speed up inference of transformers with RoPE (such as LLaMA, Mistral, PaLM, and Gemma). For these models, a large portion of the first transformer layer can be precomputed, which res…
Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting
Gradient boosting still dominates Transformers on tabular benchmarks. Our tokenizer uses a deliberately simplistic discretized vocabulary so we can highlight how even basic tokenization unlocks the power of attention on …