Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
Knowledge distillation is a critical technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student). The objective function of knowledge distillation is typically the cross-entropy between the teacher and the student's output distributions. However, for structured prediction problems, the output space is exponential in size; therefore, the cross-entropy objective becomes intractable to compute and optimize directly. In this paper, we derive a factorized form of the knowledge distillation objective for structured prediction, which is tractable for many typical choices of the teacher and student models. In particular, we show the tractability and empirical effectiveness of structural knowledge distillation between sequence labeling and dependency parsing models under four different scenarios: 1) the teacher and student share the same factorization form of the output structure scoring function; 2) the student factorization produces more fine-grained substructures than the teacher factorization; 3) the teacher factorization produces more fine-grained substructures than the student factorization; 4) the factorization forms from the teacher and the student are incompatible.
Code (1)
Tasks
Dependency ParsingKnowledge DistillationStructured PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph
In visual tasks, large teacher models capture essential features and deep information, enhancing performance. However, distilling this information into smaller student models often leads to performance loss due to struct…
Multi-Granularity Structural Knowledge Distillation for Language Model Compression
Transferring the knowledge to a small model through distillation has raised great interest in recent years. Prevailing methods transfer the knowledge derived from mono-granularity language units (e.g., token-level or sam…
Knowledge DistillationLanguage ModelingLanguage ModellingModel Compression+1On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning
Few-shot Class-Incremental Learning (FSCIL) addresses the challenges of evolving data distributions and the difficulty of data acquisition in real-world scenarios. To counteract the catastrophic forgetting typically enco…
class-incremental learningClass Incremental LearningFew-Shot Class-Incremental LearningIncremental Learning+1Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs
Distilling high-accuracy Graph Neural Networks (GNNs) to low-latency multilayer perceptions (MLPs) on graph tasks has become a hot research topic. However, conventional MLP learning relies almost exclusively on graph nod…
Knowledge DistillationContrastive Representation Distillation
Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or en…
Contrastive LearningKnowledge DistillationModel CompressionTransfer Learning