Using Active Speaker Faces for Diarization in TV shows
Speaker diarization is one of the critical components of computational media intelligence as it enables a character-level analysis of story portrayals and media content understanding. Automated audio-based speaker diarization of entertainment media poses challenges due to the diverse acoustic conditions present in media content, be it background music, overlapping speakers, or sound effects. At the same time, speaking faces in the visual modality provide complementary information and not prone to the errors seen in the audio modality. In this paper, we address the problem of speaker diarization in TV shows using the active speaker faces. We perform face clustering on the active speaker faces and show superior speaker diarization performance compared to the state-of-the-art audio-based diarization methods. We additionally report a systematic analysis of the impact of active speaker face detection quality on the diarization performance. We also observe that a moderately well-performing active speaker system could outperform the audio-based diarization systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Face ClusteringFace Detectionspeaker-diarizationSpeaker DiarizationSimilar Papers 제목 키워드 기반
Utterance-by-utterance overlap-aware neural diarization with Graph-PIT
Recent speaker diarization studies showed that integration of end-to-end neural diarization (EEND) and clustering-based diarization is a promising approach for achieving state-of-the-art performance on various tasks. Suc…
ClusteringSegmentationspeaker-diarizationSpeaker DiarizationGeodesic interpolation of frame-wise speaker embeddings for the diarization of meeting scenarios
We propose a modified teacher-student training for the extraction of frame-wise speaker embeddings that allows for an effective diarization of meeting scenarios containing partially overlapping speech. To this end, a geo…
ClusteringASoBO: Attentive Beamformer Selection for Distant Speaker Diarization in Meetings
Speaker Diarization (SD) aims at grouping speech segments that belong to the same speaker. This task is required in many speech-processing applications, such as rich meeting transcription. In this context, distant microp…
speaker-diarizationSpeaker DiarizationUnsupervised Speaker Diarization that is Agnostic to Language, Overlap-Aware, and Tuning Free
Podcasts are conversational in nature and speaker changes are frequent -- requiring speaker diarization for content understanding. We propose an unsupervised technique for speaker diarization without relying on language-…
speaker-diarizationSpeaker DiarizationDiarization-Aware Multi-Speaker Automatic Speech Recognition via Large Language Models
Multi-speaker automatic speech recognition (MS-ASR) faces significant challenges in transcribing overlapped speech, a task critical for applications like meeting transcription and conversational analysis. While serialize…
Automatic Speech Recognitionspeaker-diarizationSpeaker Diarizationspeech-recognition+1