Pretrained Language Models for Sequential Sentence Classification
As a step toward better document-level understanding, we explore classification of a sequence of sentences into their corresponding categories, a task that requires understanding sentences in context of the document. Recent successful models for this task have used hierarchical models to contextualize sentence representations, and Conditional Random Fields (CRFs) to incorporate dependencies between subsequent labels. In this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF. Specifically, we construct a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences. Our approach achieves state-of-the-art results on four datasets, including a new dataset of structured scientific abstracts.
Code (1)
Tasks
ClassificationGeneral ClassificationSentenceSentence ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts
Prevalent models based on artificial neural network (ANN) for sentence classification often classify sentences in isolation without considering the context in which sentences appear. This hampers the traditional sentence…
BenchmarkingClassificationGeneral ClassificationSentence+2Sentence Bottleneck Autoencoders from Transformer Language Models
Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems. This approach stands in contrast to autoencoders, also trained on raw tex…
DecoderDenoisingLanguage ModelingLanguage Modelling+6Generating Natural Language from Logic Expressions with Structural Representation
Incorporating logic reasoning with deep neural networks (DNNs) is an important challenge in machine learning. In this article, we study the problem of converting logical expressions into natural language. In particular, …
SentenceJointly Trained Sequential Labeling and Classification by Sparse Attention Neural Networks
Sentence-level classification and sequential labeling are two fundamental tasks in language understanding. While these two tasks are usually modeled separately, in reality, they are often correlated, for example in inten…
ClassificationGeneral Classificationintent-classificationIntent Classification+8STF: Sentence Transformer Fine-Tuning For Topic Categorization With Limited Data
Nowadays, topic classification from tweets attracts considerable research attention. Different classification systems have been suggested thanks to these research efforts. Nevertheless, they face major challenges owing t…
ClassificationSentenceTopic Classification