Self-supervised learning for audio-visual speaker diarization
Speaker diarization, which is to find the speech segments of specific speakers, has been widely used in human-centered applications such as video conferences or human-computer interaction systems. In this paper, we propose a self-supervised audio-video synchronization learning method to address the problem of speaker diarization without massive labeling effort. We improve the previous approaches by introducing two new loss functions: the dynamic triplet loss and the multinomial loss. We test them on a real-world human-computer interaction system and the results show our best model yields a remarkable gain of +8%F1-scoresas well as diarization error rate reduction. Finally, we introduce a new large scale audio-video corpus designed to fill the vacancy of audio-video datasets in Chinese.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised Learningspeaker-diarizationSpeaker DiarizationTripletVideo SynchronizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization
The scarcity of labeled audio-visual datasets is a constraint for training superior audio-visual speaker diarization systems. To improve the performance of audio-visual speaker diarization, we leverage pre-trained superv…
Decoderspeaker-diarizationSpeaker DiarizationAudio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion
Speaker diarization consists of assigning speech signals to people engaged in a dialogue. An audio-visual spatiotemporal diarization model is proposed. The model is well suited for challenging scenarios that consist of s…
Clusteringspeaker-diarizationSpeaker DiarizationVisual TrackingMulti-Input Multi-Output Target-Speaker Voice Activity Detection For Unified, Flexible, and Robust Audio-Visual Speaker Diarization
Audio-visual learning has demonstrated promising results in many classical speech tasks (e.g., speech separation, automatic speech recognition, wake-word spotting). We believe that introducing visual modality will also b…
Action DetectionActivity Detectionaudio-visual learningAutomatic Speech Recognition+5Self-supervised Representation Learning With Path Integral Clustering For Speaker Diarization
Automatic speaker diarization techniques typically involve a two-stage processing approach where audio segments of fixed duration are converted to vector representations in the first stage. This is followed by an unsuper…
ClusteringRepresentation LearningSelf-Supervised Learningspeaker-diarization+1Late Audio-Visual Fusion for In-The-Wild Speaker Diarization
Speaker diarization is well studied for constrained audios but little explored for challenging in-the-wild videos, which have more speakers, shorter utterances, and inconsistent on-screen speakers. We address this gap by…
speaker-diarizationSpeaker DiarizationSpeaker Recognition