How does BERT process disfluency?
Natural conversations are filled with disfluencies. This study investigates if and how BERT understands disfluency with three experiments: (1) a behavioural study using a downstream task, (2) an analysis of sentence embeddings and (3) an analysis of the attention mechanism on disfluency. The behavioural study shows that without fine-tuning on disfluent data, BERT does not suffer significant performance loss when presented disfluent compared to fluent inputs (exp1). Analysis on sentence embeddings of disfluent and fluent sentence pairs reveals that the deeper the layer, the more similar their representation (exp2). This indicates that deep layers of BERT become relatively invariant to disfluency. We pinpoint attention as a potential mechanism that could explain this phenomenon (exp3). Overall, the study suggests that BERT has knowledge of disfluency structure. We emphasise the potential of using BERT to understand natural utterances without disfluency removal.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceSentence EmbeddingsTask 2Similar Papers 제목 키워드 기반
Disfluency Detection for Vietnamese
In this paper, we present the first empirical study for Vietnamese disfluency detection. To conduct this study, we first create a disfluency detection dataset for Vietnamese, with manual annotations over two disfluency t…
Vietnamese Word SegmentationXLM-RTeaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection
In modern interactive speech-based systems, speech is consumed and transcribed incrementally prior to having disfluencies removed. This post-processing step is crucial for producing clean transcripts and high performance…
Machine TranslationTeaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection
In modern interactive speech-based systems speech is consumed and transcribed incrementally prior to having disfluencies removed. While this post-processing step is crucial for producing clean transcripts and high perfor…
Machine TranslationTranslationDisfluency Detection with Unlabeled Data and Small BERT Models
Disfluency detection models now approach high accuracy on English text. However, little exploration has been done in improving the size and inference time of the model. At the same time, automatic speech recognition (ASR…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Data AugmentationDomain Adaptation+2Streaming Joint Speech Recognition and Disfluency Detection
Disfluency detection has mainly been solved in a pipeline approach, as post-processing of speech recognition. In this study, we propose Transformer-based encoder-decoder models that jointly solve speech recognition and d…
DecoderLanguage Modellingspeech-recognitionSpeech Recognition