Investigating Text Shortening Strategy in BERT: Truncation vs Summarization
The parallelism of Transformer-based models comes at the cost of their input max-length. Some studies proposed methods to overcome this limitation, but none of them reported the effectiveness of summarization as an alternative. In this study, we investigate the performance of document truncation and summarization in text classification tasks. Each of the two was investigated with several variations. This study also investigated how close their performances are to the performance of full-text. We used a dataset of summarization tasks based on Indonesian news articles (IndoSum) to do classification tests. This study shows how the summaries outperform the majority of truncation method variations and lose to only one. The best strategy obtained in this study is taking the head of the document. The second is extractive summarization. This study explains what happened to the result, leading to further research in order to exploit the potential of document summarization as a shortening alternative. The code and data used in this work are publicly available in https://github.com/mirzaalimm/TruncationVsSummarization.
Code (1)
Tasks
ArticlesDocument SummarizationExtractive Summarizationtext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Ranked List Truncation for Large Language Model-based Re-Ranking
We study ranked list truncation (RLT) from a novel "retrieve-then-re-rank" perspective, where we optimize re-ranking by truncating the retrieved list (i.e., trim re-ranking candidates). RLT is crucial for re-ranking as i…
Language ModelingLanguage ModellingLarge Language ModelRe-Ranking+1Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT
Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, deploying these models can be prohibit…
InformativenessA Unified Retrieval Framework with Document Ranking and EDU Filtering for Multi-document Summarization
In the field of multi-document summarization (MDS), transformer-based models have demonstrated remarkable success, yet they suffer an input length limitation. Current methods apply truncation after the retrieval process …
Document RankingDocument SummarizationMulti-Document SummarizationRetrievalClosing the Curious Case of Neural Text Degeneration
Despite their ubiquity in language generation, it remains unknown why truncation sampling heuristics like nucleus sampling are so effective. We provide a theoretical explanation for the effectiveness of the truncation sa…
Text GenerationRebuildable biochronometer: inferences and hypothesis on eukaryotic timing system
The biochronometers used to keep time in eukaryotes include short-period biochronometer (SPB) and long-period biochronometer (LPB). Because the circadian clock reflects the biological time rhythm of a day, it is consider…
Rhythm