Zero-shot Conversational Summarization Evaluations with small Large Language Models
Large Language Models (LLMs) exhibit powerful summarization abilities. However, their capabilities on conversational summarization remains under explored. In this work we evaluate LLMs (approx. 10 billion parameters) on conversational summarization and showcase their performance on various prompts. We show that the summaries generated by models depend on the instructions and the performance of LLMs vary with different instructions sometimes resulting steep drop in ROUGE scores if prompts are not selected carefully. We also evaluate the models with human evaluations and discuss the limitations of the models on conversational summarization
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations
The rapid advancements in Large Language Models (LLMs) have enabled their adoption in real-world industrial scenarios for various natural language processing tasks. However, the high inference cost of large-scale LLMs ma…
Zero-shot GeneralizationReading ComprehensionMeeting SummarizationDomain AdaptationRestructuring Conversations using Discourse Relations for Zero-shot Abstractive Dialogue Summarization
Dialogue summarization is a challenging problem due to the informal and unstructured nature of conversational data. Recent advances in abstractive summarization have been focused on data-hungry neural models and adapting…
Abstractive Dialogue SummarizationAbstractive Text SummarizationDocument SummarizationForumSum: A Multi-Speaker Conversation Summarization Dataset
Abstractive summarization quality had large improvements since recent language pretraining techniques. However, currently there is a lack of datasets for the growing needs of conversation summarization applications. Thus…
Abstractive Text SummarizationConversation SummarizationNeural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
In zero-shot multilingual extractive text summarization, a model is typically trained on English summarization dataset and then applied on summarization datasets of other languages. Given English gold summaries and docum…
Extractive SummarizationExtractive Text SummarizationSentenceText SummarizationInheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT
While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications. Conversely…