Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning

2018-11-07 · Xin Wang, Jiawei Wu, Da Zhang, Yu Su, William Yang Wang

Although promising results have been achieved in video captioning, existing models are limited to the fixed inventory of activities in the training corpus, and do not generalize to open vocabulary scenarios. Here we introduce a novel task, zero-shot video captioning, that aims at describing out-of-domain videos of unseen activities. Videos of different activities usually require different captioning strategies in many aspects, i.e. word selection, semantic construction, and style expression etc, which poses a great challenge to depict novel activities without paired training data. But meanwhile, similar activities share some of those aspects in common. Therefore, We propose a principled Topic-Aware Mixture of Experts (TAMoE) model for zero-shot video captioning, which learns to compose different experts based on different topic embeddings, implicitly transferring the knowledge learned from seen activities to unseen ones. Besides, we leverage external topic-related text corpus to construct the topic embedding for each activity, which embodies the most relevant semantic vectors within the topic. Empirical results not only validate the effectiveness of our method in utilizing semantic knowledge for video captioning, but also show its strong generalization ability when describing novel activities.

📄 PDF Abstract BibTeX arXiv:1811.02765

Code (0)

등록된 구현이 없습니다.

Tasks

Mixture-of-ExpertsVideo Captioning

Fine-Grained Zero-Shot Learning with Attribute-Centric Representations

2025-12-13 · Zhi Chen, Jingcai Guo, Taotao Cai, Yuxiang Cai arxiv

Recognizing unseen fine-grained categories demands a model that can distinguish subtle visual differences. This is typically achieved by transferring visual-attribute relationships from seen classes to unseen classes. Th…

Representation Learning Zero-Shot Learning

WMoE-CLIP: Wavelet-Enhanced Mixture-of-Experts Prompt Learning for Zero-Shot Anomaly Detection

2026-03-06 · Peng Chen, Chao Huang arxiv

Vision-language models have recently shown strong generalization in zero-shot anomaly detection (ZSAD), enabling the detection of unseen anomalies without task-specific supervision. However, existing approaches typically…

Anomaly Detection

HOMOE: A Memory-Based and Composition-Aware Framework for Zero-Shot Learning with Hopfield Network and Soft Mixture of Experts

2023-11-23 · Do Huu Dat, Po Yuan Mao, Tien Hoang Nguyen, Wray Buntine 외

Compositional Zero-Shot Learning (CZSL) has emerged as an essential paradigm in machine learning, aiming to overcome the constraints of traditional zero-shot learning by incorporating compositional thinking into its meth…

Compositional Zero-Shot Learning Mixture-of-Experts Zero-Shot Learning

Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs

2026-05-28 · Zhibo Zhang, Yuxi Li, Zhen Ouyang, Ling Shi 외 arxiv

Mixture-of-Experts (MoE) LLMs rely on sparse, router-driven expert activation, yet how safety alignment interacts with routed expert specialization remains underexplored. A common intuition is that safety behavior may be…

GZSL-MoE: Apprentissage G{é}n{é}ralis{é} Z{é}ro-Shot bas{é} sur le M{é}lange d'Experts pour la Segmentation S{é}mantique de Nuages de Points 3DAppliqu{é} {à} un Jeu de Donn{é}es d'Environnement de Collaboration Humain-Robot

2025-09-23 · Ahed Alboody arxiv

Generative Zero-Shot Learning approach (GZSL) has demonstrated significant potential in 3D point cloud semantic segmentation tasks. GZSL leverages generative models like GANs or VAEs to synthesize realistic features (rea…

Generalized Zero-Shot Learning 3D Semantic Segmentation