Layer-wise Guided Training for BERT: Learning Incrementally Refined Document Representations

2020-10-12 · EMNLP (spnlp) 2020 11 · Nikolaos Manginas, Ilias Chalkidis, Prodromos Malakasiotis

Although BERT is widely used by the NLP community, little is known about its inner workings. Several attempts have been made to shed light on certain aspects of BERT, often with contradicting conclusions. A much raised concern focuses on BERT's over-parameterization and under-utilization issues. To this end, we propose o novel approach to fine-tune BERT in a structured manner. Specifically, we focus on Large Scale Multilabel Text Classification (LMTC) where documents are assigned with one or more labels from a large predefined set of hierarchically organized labels. Our approach guides specific BERT layers to predict labels from specific hierarchy levels. Experimenting with two LMTC datasets we show that this structured fine-tuning approach not only yields better classification results but also leads to better parameter utilization.

📄 PDF Abstract BibTeX arXiv:2010.05763

Code (0)

등록된 구현이 없습니다.

Tasks

General ClassificationMultilabel Text Classificationtext-classification Text Classification

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

WordPiece 설명 없음

Adam 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Knowledge Transfer Pre-training

2015-06-07 · Zhiyuan Tang, Dong Wang, Yiqiao Pan, Zhiyong Zhang

Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. Th…

speech-recognition Speech Recognition Transfer Learning

Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup

2020-11-27 · Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li 외

Pre-trained language models, such as BERT, have achieved significant accuracy gain in many natural language processing tasks. Despite its effectiveness, the huge number of parameters makes training a BERT model computati…

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

2019-04-01 · ICLR 2020 1 · Yang You, Jing Li, Sashank Reddi, Jonathan Hseu 외

Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most pro…

Question Answering Stochastic Optimization

MaxPoolBERT: Enhancing BERT Classification via Layer- and Token-Wise Aggregation

2025-05-21 · Maike Behrendt, Stefan Sylvius Wagner, Stefan Harmeling

The [CLS] token in BERT is commonly used as a fixed-length representation for classification tasks, yet prior work has shown that both other tokens and intermediate layers encode valuable contextual information. In this …

Let the Model Learn to Feel: Mode-Guided Tonality Injection for Symbolic Music Emotion Recognition

2025-12-15 · Haiying Xia, Zhongyi Huang, Yumei Tan, Shuxiang Song arxiv

Music emotion recognition is a key task in symbolic music understanding (SMER). Recent approaches have shown promising results by fine-tuning large-scale pre-trained models (e.g., MIDIBERT, a benchmark in symbolic music …

Emotion Recognition