MOD: A Deep Mixture Model with Online Knowledge Distillation for Large Scale Video Temporal Concept Localization

2019-10-27 · Rongcheng Lin, Jing Xiao, Jianping Fan

In this paper, we present and discuss a deep mixture model with online knowledge distillation (MOD) for large-scale video temporal concept localization, which is ranked 3rd in the 3rd YouTube-8M Video Understanding Challenge. Specifically, we find that by enabling knowledge sharing with online distillation, fintuning a mixture model on a smaller dataset can achieve better evaluation performance. Based on this observation, in our final solution, we trained and fintuned 12 NeXtVLAD models in parallel with a 2-layer online distillation structure. The experimental results show that the proposed distillation structure can effectively avoid overfitting and shows superior generalization performance. The code is publicly available at: https://github.com/linrongc/solution_youtube8m_v3

📄 PDF Abstract BibTeX arXiv:1910.12295

Code (1)

linrongc/solution_youtube8m_v3 공식 구현 tf

Tasks

Knowledge DistillationVideo Understanding

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

2026-06-09 · Xinyue Peng, Yi Qian, Jiaojiao Lin, Wenjian Shao 외 arxiv

As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for di…

Knowledge Distillation Mathematical Reasoning

Multi scale Feature Extraction and Fusion for Online Knowledge Distillation

2022-06-16 · Panpan Zou, Yinglei Teng, Tao Niu

Online knowledge distillation conducts knowledge transfer among all student models to alleviate the reliance on pre-trained models. However, existing online methods rely heavily on the prediction distributions and neglec…

Knowledge Distillation Transfer Learning

Efficient Cross-Architecture Knowledge Transfer for Large-Scale Online User Response Prediction

2026-02-02 · Yucheng Wu, Yuekui Yang, Hongzheng Li, Anan Liu 외 arxiv

Deploying new architectures in large-scale user response prediction systems incurs high model switching costs due to expensive retraining on massive historical data and performance degradation under data retention constr…

Knowledge Distillation

Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillation

2026-02-02 · Seonghyeon Park, Jewon Yeom, Jaewon Sok, Jeongjae Park 외 arxiv

Quantifying uncertainty in Large Language Models (LLMs) is essential for mitigating hallucinations and enabling risk-aware deployment in safety-critical tasks. However, estimating Epistemic Uncertainty(EU) via Deep Ensem…

Knowledge Distillation

DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings

2021-12-10 · Chaochen Gao, Xing Wu, Peng Wang, Jue Wang 외

Large-scale contrastive learning models can learn very informative sentence embeddings, but are hard to serve online due to the huge model size. Therefore, they often play the role of "teacher", transferring abilities to…

Contrastive Learning Knowledge Distillation Semantic Textual Similarity Sentence+4