Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification
Distance Metric Learning (DML) has typically dominated the audio-visual speaker verification problem space, owing to strong performance in new and unseen classes. In our work, we explored multitask learning techniques to further enhance DML, and show that an auxiliary task with even weak labels can increase the quality of the learned speaker representation without increasing model complexity during inference. We also extend the Generalized End-to-End Loss (GE2E) to multimodal inputs and demonstrate that it can achieve competitive performance in an audio-visual space. Finally, we introduce AV-Mixup, a multimodal augmentation technique during training time that has shown to reduce speaker overfit. Our network achieves state of the art performance for speaker verification, reporting 0.244%, 0.252%, 0.441% Equal Error Rate (EER) on the VoxCeleb1-O/E/H test sets, which is to our knowledge, the best published results on VoxCeleb1-E and VoxCeleb1-H.
Code (0)
등록된 구현이 없습니다.
Tasks
Metric LearningMulti-Task LearningSpeaker VerificationSimilar Papers 제목 키워드 기반
Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup
Mixup is a recent regularizer for current deep classification networks. Through training a neural network on convex combinations of pairs of examples and their labels, it imposes locally linear constraints on the model's…
text-classificationText ClassificationCroMo-Mixup: Augmenting Cross-Model Representations for Continual Self-Supervised Learning
Continual self-supervised learning (CSSL) learns a series of tasks sequentially on the unlabeled data. Two main challenges of continual learning are catastrophic forgetting and task confusion. While CSSL problem has been…
class-incremental learningClass Incremental LearningContinual LearningContinual Self-Supervised Learning+2DualMix: Unleashing the Potential of Data Augmentation for Online Class-Incremental Learning
Online Class-Incremental (OCI) learning has sparked new approaches to expand the previously trained model knowledge from sequentially arriving data streams with new classes. Unfortunately, OCI learning can suffer from ca…
class-incremental learningClass Incremental LearningData AugmentationIncremental LearningIntra-Cluster Mixup: An Effective Data Augmentation Technique for Complementary-Label Learning
In this paper, we investigate the challenges of complementary-label learning (CLL), a specialized form of weakly-supervised learning (WSL) where models are trained with labels indicating classes to which instances do not…
Data AugmentationMixup-CAM: Weakly-supervised Semantic Segmentation via Uncertainty Regularization
Obtaining object response maps is one important step to achieve weakly-supervised semantic segmentation using image-level labels. However, existing methods rely on the classification task, which could result in a respons…
ClassificationData AugmentationGeneral ClassificationObject+3