BER: Balanced Error Rate For Speaker Diarization
DER is the primary metric to evaluate diarization performance while facing a dilemma: the errors in short utterances or segments tend to be overwhelmed by longer ones. Short segments, e.g., yes' or no,' still have semantic information. Besides, DER overlooks errors in less-talked speakers. Although JER balances speaker errors, it still suffers from the same dilemma. Considering all those aspects, duration error, segment error, and speaker-weighted error constituting a complete diarization evaluation, we propose a Balanced Error Rate (BER) to evaluate speaker diarization. First, we propose a segment-level error rate (SER) via connected sub-graphs and adaptive IoU threshold to get accurate segment matching. Second, to evaluate diarization in a unified way, we adopt a speaker-specific harmonic mean between duration and segment, followed by a speaker-weighted average. Third, we analyze our metric via the modularized system, EEND, and the multi-modal method on real datasets. SER and BER are publicly available at https://github.com/X-LANCE/BER.
Code (2)
Tasks
speaker-diarizationSpeaker DiarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Investigating Confidence Estimation Measures for Speaker Diarization
Speaker diarization systems segment a conversation recording based on the speakers' identity. Such systems can misclassify the speaker of a portion of audio due to a variety of factors, such as speech pattern variation, …
speaker-diarizationSpeaker Diarizationspeech-recognitionSpeech RecognitionDomain-Aware Speaker Diarization On African-Accented English
This study examines domain effects in speaker diarization for African-accented English. We evaluate multiple production and open systems on general and clinical dialogues under a strict DER protocol that scores overlap. …
Speaker DiarizationDomain AdaptationTalTech-IRIT-LIS Speaker and Language Diarization Systems for DISPLACE 2024
This paper describes the submissions of team TalTech-IRIT-LIS to the DISPLACE 2024 challenge. Our team participated in the speaker diarization and language diarization tracks of the challenge. In the speaker diarization …
speaker-diarizationSpeaker DiarizationSpeech SeparationDiaCorrect: End-to-end error correction for speaker diarization
In recent years, speaker diarization has attracted widespread attention. To achieve better performance, some studies propose to diarize speech in multiple stages. Although these methods might bring additional benefits, m…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speaker-diarizationSpeaker Diarization+3End-to-End Neural Speaker Diarization with Permutation-Free Objectives
In this paper, we propose a novel end-to-end neural-network-based speaker diarization method. Unlike most existing methods, our proposed method does not have separate modules for extraction and clustering of speaker repr…
ClusteringDomain AdaptationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATION+2