Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
This work presents a novel objective function for the unsupervised training of neural network sentence encoders. It exploits signals from paragraph-level discourse coherence to train these models to understand text. Our objective is purely discriminative, allowing us to train models many times faster than was possible under prior methods, and it yields models which perform well in extrinsic evaluations.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSentenceSimilar Papers 제목 키워드 기반
Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
Recent models for unsupervised representation learning of text have employed a number of techniques to improve contextual word representations but have put little focus on discourse-level representations. We propose CONP…
Common Sense ReasoningNatural Language InferenceReading ComprehensionRepresentation Learning+2DEPTH: Discourse Education through Pre-Training Hierarchically
Language Models (LMs) struggle with linguistic understanding at the discourse level, even though discourse patterns such as coherence, cohesion, and narrative flow are prevalent in their pre-training data. To improve the…
DecoderNatural Language UnderstandingSentenceEvaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
Prior work on pretrained sentence embeddings and benchmarks focus on the capabilities of stand-alone sentences. We propose DiscoEval, a test suite of tasks to evaluate whether sentence representations include broader con…
SentenceSentence EmbeddingsMining Discourse Markers for Unsupervised Sentence Representation Learning
Current state of the art systems in NLP heavily rely on manually annotated datasets, which are expensive to construct. Very little work adequately exploits unannotated data -- such as discourse markers between sentences …
Relation ClassificationRepresentation LearningSentenceSentence EmbeddingsLong Text Generation by Modeling Sentence-Level and Discourse-Level Coherence
Generating long and coherent text is an important but challenging task, particularly for open-ended language generation tasks such as story generation. Despite the success in modeling intra-sentence coherence, existing g…
DecoderSemantic SimilaritySemantic Textual SimilaritySentence+2