Unifying distillation and privileged information
Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines. This paper unifies these two techniques into generalized distillation, a framework to learn from multiple machines and data representations. We provide theoretical and causal insight about the inner workings of generalized distillation, extend it to unsupervised, semisupervised and multitask learning scenarios, and illustrate its efficacy on a variety of numerical simulations on both synthetic and real-world data.
Code (1)
Similar Papers 제목 키워드 기반
Rethinking On-Policy Self-Distillation for Thinking Models
Self-distillation is a promising recipe for self-improvement in language models. In this setting, a model can serve as its own teacher when given privileged information, such as a solution to a math problem. This seems e…
Privileged Information Dropout in Reinforcement Learning
Using privileged information during training can improve the sample efficiency and performance of machine learning systems. This paradigm has been applied to reinforcement learning (RL), primarily in the form of distilla…
Formreinforcement-learningReinforcement LearningReinforcement Learning (RL)Privileged Prior Information Distillation for Image Matting
Performance of trimap-free image matting methods is limited when trying to decouple the deterministic and undetermined regions, especially in the scenes where foregrounds are semantically ambiguous, chromaless, or high t…
Image Matting$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data
Deep search agents have emerged as a promising paradigm for addressing complex information-seeking tasks, but their training remains challenging due to sparse rewards, weak credit assignment, and limited labeled data. Se…
Privileged Knowledge Distillation for Online Action Detection
Online Action Detection (OAD) in videos is proposed as a per-frame labeling task to address the real-time prediction tasks that can only obtain the previous and current video frames. This paper presents a novel learning-…
Action DetectionKnowledge DistillationOnline Action Detection