Alignment-Uniformity aware Representation Learning for Zero-shot Video Classification

2022-03-29 · CVPR 2022 1 · Shi Pu, Kaili Zhao, Mao Zheng

Most methods tackle zero-shot video classification by aligning visual-semantic representations within seen classes, which limits generalization to unseen classes. To enhance model generalizability, this paper presents an end-to-end framework that preserves alignment and uniformity properties for representations on both seen and unseen classes. Specifically, we formulate a supervised contrastive loss to simultaneously align visual-semantic features (i.e., alignment) and encourage the learned features to distribute uniformly (i.e., uniformity). Unlike existing methods that only consider the alignment, we propose uniformity to preserve maximal-info of existing features, which improves the probability that unobserved features fall around observed data. Further, we synthesize features of unseen classes by proposing a class generator that interpolates and extrapolates the features of seen classes. Besides, we introduce two metrics, closeness and dispersion, to quantify the two properties and serve as new measurements of model generalizability. Experiments show that our method significantly outperforms SoTA by relative improvements of 28.1% on UCF101 and 27.0% on HMDB51. Code is available.

📄 PDF Abstract BibTeX arXiv:2203.15381

Code (1)

ShipuLoveMili/CVPR2022-AURL 공식 구현 pytorch

Tasks

Representation Learning Video Classification Zero-Shot Action Recognition

Methods 이 논문이 사용한 방법론

Supervised Contrastive Loss 설명 없음

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Geodesic Multi-Modal Mixup for Robust Fine-Tuning

2022-03-08 · NeurIPS 2023 11 · Changdae Oh, Junhyuk So, Hoyoon Byun, Yongtaek Lim 외

Pre-trained multi-modal models, such as CLIP, provide transferable embeddings and show promising results in diverse applications. However, the analysis of learned multi-modal embeddings is relatively unexplored, and the …

Image Captioning zero-shot-classification Zero-Shot Learning

Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruption

2025-05-19 · Kazuki Adachi, Shin'ya Yamaguchi, Tomoki Hamagami

Pre-trained vision-language models such as contrastive language-image pre-training (CLIP) have demonstrated a remarkable generalizability, which has enabled a wide range of applications represented by zero-shot classific…

Knowledge Distillation Test-time Adaptation zero-shot-classification Zero-Shot Learning

Conditional Alignment and Uniformity for Contrastive Learning with Continuous Proxy Labels

2021-11-10 · Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis 외

Contrastive Learning has shown impressive results on natural and medical images, without requiring annotated data. However, a particularity of medical images is the availability of meta-data (such as age or sex) that can…

Contrastive Learning Linear evaluation

It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap

2024-05-28 · Abrar Fahim, Alex Murphy, Alona Fyshe

Multi-modal contrastive models such as CLIP achieve state-of-the-art performance in zero-shot classification by embedding input images and texts on a joint representational space. Recently, a modality gap has been report…

image-classification Image Classification zero-shot-classification Zero-Shot Image Classification+1

Toward a Better Understanding of Loss Functions for Collaborative Filtering

2023-08-11 · Seongmin Park, Mincheol Yoon, Jae-woong Lee, Hogun Park 외

Collaborative filtering (CF) is a pivotal technique in modern recommender systems. The learning process of CF models typically consists of three components: interaction encoder, loss function, and negative sampling. Alth…

Collaborative Filtering Recommendation Systems