Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study
Text classification is an area of research which has been studied over the years in Natural Language Processing (NLP). Adapting NLP to multiple domains has introduced many new challenges for text classification and one of them is long document classification. While state-of-the-art transformer models provide excellent results in text classification, most of them have limitations in the maximum sequence length of the input sequence. The majority of the transformer models are limited to 512 tokens, and therefore, they struggle with long document classification problems. In this research, we explore on employing Model Fusing for long document classification while comparing the results with well-known BERT and Longformer architectures.
Code (1)
Tasks
ClassificationDocument Classificationtext-classificationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fusing Temporal Graphs into Transformers for Time-Sensitive Question Answering
Answering time-sensitive questions from long documents requires temporal reasoning over the times in questions and documents. An important open question is whether large language models can perform such reasoning solely …
Question AnsweringTemporal Information ExtractionRevisiting Transformer-based Models for Long Document Classification
The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficien…
ClassificationDocument Classificationtext-classificationText ClassificationRevisiting Transformer-based Models for Long Document Classification
The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficie…
ClassificationDocument Classificationtext-classificationText ClassificationEfficient Classification of Long Documents Using Transformers
Several methods have been proposed for classifying long textual documents using Transformers. However, there is a lack of consensus on a benchmark to enable a fair comparison among different approaches. In this paper, we…
ClassificationDocument ClassificationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONAn Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification
Non-hierarchical sparse attention Transformer-based models, such as Longformer and Big Bird, are popular approaches to working with long documents. There are clear benefits to these approaches compared to the original Tr…
Document ClassificationGPU