DocBERT: BERT for Document Classification
We present, to our knowledge, the first application of BERT to document classification. A few characteristics of the task might lead one to think that BERT is not the most appropriate model: syntactic structures matter less for content categories, documents can often be longer than typical BERT input, and documents often have multiple labels. Nevertheless, we show that a straightforward classification model using BERT is able to achieve the state of the art across four popular datasets. To address the computational expense associated with BERT inference, we distill knowledge from BERT-large to small bidirectional LSTMs, reaching BERT-base parity on multiple datasets using 30x fewer parameters. The primary contribution of our paper is improved baselines that can provide the foundation for future work.
Code (3)
Tasks
ClassificationDocument ClassificationGeneral ClassificationSentiment AnalysisText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring the Limits of Simple Learners in Knowledge Distillation for Document Classification with DocBERT
Fine-tuned variants of BERT are able to achieve state-of-the-art accuracy on many natural language processing tasks, although at significant computational costs. In this paper, we verify BERT{'}s effectiveness for docume…
Document ClassificationGeneral ClassificationKnowledge DistillationModel CompressionBenchmarking LLMs for Predictive Applications in the Intensive Care Units
With the advent of LLMs, various tasks across the natural language processing domain have been transformed. However, their application in predictive tasks remains less researched. This study compares large language model…
A Sentence-level Hierarchical BERT Model for Document Classification with Limited Labelled Data
Training deep learning models with limited labelled data is an attractive scenario for many NLP tasks, including document classification. While with the recent emergence of BERT, deep learning language models can achieve…
ClassificationDocument ClassificationSentenceEffectively Leveraging BERT for Legal Document Classification
Bidirectional Encoder Representations from Transformers (BERT) has achieved state-of-the-art performances on several text classification tasks, such as GLUE and sentiment analysis. Recent work in the legal domain started…
Binary ClassificationClassificationDocument ClassificationSentiment Analysis+2Application of Mix-Up Method in Document Classification Task Using BERT
The mix-up method (Zhang et al., 2017), one of the methods for data augmentation, is known to be easy to implement and highly effective. Although the mix-up method is intended for image identification, it can also be app…
ClassificationData AugmentationDocument ClassificationSentence