Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues
Discourse processing suffers from data sparsity, especially for dialogues. As a result, we explore approaches to build discourse structures for dialogues, based on attention matrices from Pre-trained Language Models (PLMs). We investigate multiple tasks for fine-tuning and show that the dialogue-tailored Sentence Ordering task performs best. To locate and exploit discourse information in PLMs, we propose an unsupervised and a semi-supervised method. Our proposals achieve encouraging results on the STAC corpus, with F1 scores of 57.2 and 59.3 for unsupervised and semi-supervised methods, respectively. When restricted to projective trees, our scores improved to 63.3 and 68.1.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceSentence OrderingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models
With a growing number of BERTology work analyzing different components of pre-trained language models, we extend this line of research through an in-depth analysis of discourse information in pre-trained and fine-tuned l…
Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models
In this paper, we extend the line of BERTology work by focusing on the important, yet less explored, alignment of pre-trained and fine-tuned PLMs with large-scale discourse structures. We propose a novel approach to infe…
Labeling Explicit Discourse Relations using Pre-trained Language Models
Labeling explicit discourse relations is one of the most challenging sub-tasks of the shallow discourse parsing where the goal is to identify the discourse connectives and the boundaries of their arguments. The state-of-…
Discourse ParsingAssessing Discourse Relations in Language Generation from GPT-2
Recent advances in NLP have been attributed to the emergence of large-scale pre-trained language models. GPT-2, in particular, is suited for generation tasks given its left-to-right language modeling objective, yet the l…
Language ModelingLanguage ModellingText GenerationvalidInstruct-SCTG: Guiding Sequential Controlled Text Generation through Instructions
Instruction-tuned large language models have shown remarkable performance in aligning generated text with user intentions across various tasks. However, maintaining human-like discourse structure in the generated text re…
ArticlesText Generation