MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learning
Data quality stands at the forefront of deciding the effectiveness of video-language representation learning. However, video-text pairs in previous data typically do not align perfectly with each other, which might lead to video-language representations that do not accurately reflect cross-modal semantics. Moreover, previous data also possess an uneven distribution of concepts, thereby hampering the downstream performance across unpopular subjects. To address these problems, we propose MAMA, a new approach to learning video-language representations by utilizing a contrastive objective with a subtractive angular margin to regularize cross-modal representations in their effort to reach perfect similarity. Furthermore, to adapt to the non-uniform concept distribution, MAMA utilizes a multi-layer perceptron (MLP)-parameterized weighting function that maps loss values to sample weights which enable dynamic adjustment of the model's focus throughout the training. With the training guided by a small amount of unbiased meta-data and augmented by video-text data generated by large vision-language model, MAMA improves video-language representations and achieve superior performances on commonly used video question answering and text-video retrieval datasets. The code, model, and data have been made available at https://nguyentthong.github.io/MAMA.
Code (1)
Tasks
Language ModelingLanguage ModellingQuestion AnsweringRepresentation LearningVideo Question AnsweringVideo RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems
LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design o…
Self-supervised learning of audio representations using angular contrastive loss
In Self-Supervised Learning (SSL), various pretext tasks are designed for learning feature representations through contrastive loss. However, previous studies have shown that this loss is less tolerant to semantically si…
Contrastive LearningSelf-Supervised LearningKDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning
Previous work on multimodal sentence embedding has proposed multimodal contrastive learning and achieved promising results. However, by taking the rest of the batch as negative samples without reviewing when forming cont…
Contrastive LearningKnowledge DistillationSemantic Textual SimilaritySentence+4Discriminative Speaker Representation via Contrastive Learning with Class-Aware Attention in Angular Space
The challenges in applying contrastive learning to speaker verification (SV) are that the softmax-based contrastive loss lacks discriminative power and that the hard negative pairs can easily influence learning. To overc…
Contrastive LearningSpeaker VerificationPrivacy Vulnerabilities in Marginals-based Synthetic Data
When acting as a privacy-enhancing technology, synthetic data generation (SDG) aims to maintain a resemblance to the real data while excluding personally-identifiable information. Many SDG algorithms provide robust diffe…
Inference AttackMembership Inference AttackSynthetic Data Generation