BagBERT: BERT-based bagging-stacking for multi-topic classification

2021-11-10 · Loïc Rakotoson, Charles Letaillieur, Sylvain Massip, Fréjus Laleye

This paper describes our submission on the COVID-19 literature annotation task at Biocreative VII. We proposed an approach that exploits the knowledge of the globally non-optimal weights, usually rejected, to build a rich representation of each label. Our proposed approach consists of two stages: (1) A bagging of various initializations of the training data that features weakly trained weights, (2) A stacking of heterogeneous vocabulary models based on BERT and RoBERTa Embeddings. The aggregation of these weak insights performs better than a classical globally efficient model. The purpose is the distillation of the richness of knowledge to a simpler and lighter model. Our system obtains an Instance-based F1 of 92.96 and a Label-based micro-F1 of 91.35.

📄 PDF Abstract BibTeX arXiv:2111.05808

Code (1)

opscidia/bagbert 공식 구현 tf

Tasks

Classification Ensemble LearningTopic Classification

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

WordPiece 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Weight Decay 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Stacking and stability

2019-01-26 · Nino Arsov, Martin Pavlovski, Ljupco Kocarev

Stacking is a general approach for combining multiple models toward greater predictive accuracy. It has found various application across different domains, ensuing from its meta-learning nature. Our understanding, nevert…

Meta-Learning

Stacking-Enhanced Bagging Ensemble Learning for Breast Cancer Classification with CNN

2024-07-15 · Peihceng Wu, Runze Ma, Teoh Teik Toe

This paper proposes a CNN classification network based on Bagging and stacking ensemble learning methods for breast cancer classification. The model was trained and tested on the public dataset of DDSM. The model is capa…

Binary Classification Cancer Classification Classification Ensemble Learning

BagStacking: An Integrated Ensemble Learning Approach for Freezing of Gait Detection in Parkinson's Disease

2024-02-24 · Seffi Cohen, Lior Rokach

This paper introduces BagStacking, a novel ensemble learning method designed to enhance the detection of Freezing of Gait (FOG) in Parkinson's Disease (PD) by using a lower-back sensor to track acceleration. Building on …

Ensemble Learning

BoostingBERT:Integrating Multi-Class Boosting into BERT for NLP Tasks

2020-09-13 · Tongwen Huang, Qingyun She, Junlin Zhang

As a pre-trained Transformer model, BERT (Bidirectional Encoder Representations from Transformers) has achieved ground-breaking performance on multiple NLP tasks. On the other hand, Boosting is a popular ensemble learnin…

Ensemble Learning Knowledge Distillation

Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems

2019-08-14 · Mohsen Shahhosseini, Guiping Hu, Hieu Pham

Aggregating multiple learners through an ensemble of models aim to make better predictions by capturing the underlying distribution of the data more accurately. Different ensembling methods, such as bagging, boosting, an…

BIG-bench Machine Learning regression