Towards End-to-end Speech-to-text Summarization
Speech-to-text (S2T) summarization is a time-saving technique for filtering and keeping up with the broadcast news uploaded online on a daily basis. The rise of large language models from deep learning with impressive text generation capabilities has placed the research focus on summarization systems that produce paraphrased compact versions of the document content, also known as abstractive summaries. End-to-end (E2E) modelling of S2T abstractive summarization is a promising approach that offers the possibility of generating rich latent representations that leverage non-verbal and acoustic information, as opposed to the use of only linguistic information from automatically generated transcripts in cascade systems. However, the few literature on E2E modelling of this task fails on exploring different domains, namely broadcast news, which is challenging domain where large and diversified volumes of data are presented to the user every day. We model S2T summarization both with a cascade and an E2E system for a corpus of broadcast news in French. Our novel E2E model leverages external data by resorting to transfer learning from a pre-trained T2T summarizer. Experiments show that both our cascade and E2E abstractive summarizers are stronger than an extractive baseline. However, the performance of the E2E model still lies behind the cascade one, which is object of an extensive analysis that includes future directions to close that gap.
Code (1)
Tasks
Abstractive Text SummarizationSpeech-to-TextText GenerationText SummarizationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Speech Summarization using Restricted Self-Attention
Speech summarization is typically performed by using a cascade of speech recognition and text summarization models. End-to-end modeling of speech summarization models is challenging due to memory and compute constraints …
Document Summarizationspeech-recognitionSpeech RecognitionSpoken Language Understanding+1ESSumm: Extractive Speech Summarization from Untranscribed Meeting
In this paper, we propose a novel architecture for direct extractive speech-to-speech summarization, ESSumm, which is an unsupervised model without dependence on intermediate transcribed text. Different from previous met…
speech-recognitionSpeech RecognitionAdvancing Speech Summarization in Multi-modal LLMs with Reinforcement Learning
Speech summarization is a critical component of spoken content understanding, particularly in the era of rapidly growing spoken and audiovisual data. Recent advances in multi-modal large language models (MLLMs), leveragi…
Zero-shot GeneralizationReinforcement LearningSummarizing Speech: A Comprehensive Survey
Speech summarization has become an essential tool for efficiently managing and accessing the growing volume of spoken and audiovisual content. However, despite its increasing importance, speech summarization remains loos…
Meeting Summarizationspeech-recognitionSpeech RecognitionSurvey+1LIA-RAG: a system based on graphs and divergence of probabilities applied to Speech-To-Text Summarization
This paper aims to introduces a new algorithm for automatic speech-to-text summarization based on statistical divergences of probabilities and graphs. The input is a text from speech conversations with noise, and the out…
RAGSpeech-to-TextText Summarization