Analysis of Alignment Phenomenon in Simple Teacher-student Networks with Finite Width
Recent theoretical analysis suggests that ultra-wide neural networks always converge to global minima near the initialization under first order methods. However, the convergence property of neural networks with finite width could be very different. The simplest experiment with two-layer teacher-student networks shows that the input weights of student neurons eventually align with one of the teacher neurons. This suggests a distinct convergence nature for ``not-too-wide'' neural networks that there might not be any local minima near the initialization. As the theoretical justification, we prove that under the most basic settings, all student neurons must align with the teacher neuron at any local minima. The methodology is extendable to more general cases, where the proof can be reduced to analyzing the properties of a special class of functions that we call {\em Angular Distance (AD) function}. Finally, we demonstrate that these properties can be easily verified numerically.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Rethinking the Knowledge Distillation From the Perspective of Model Calibration
Recent years have witnessed dramatically improvements in the knowledge distillation, which can generate a compact student model for better efficiency while retaining the model effectiveness of the teacher model. Previous…
Knowledge DistillationOn the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
Weak-to-strong generalization, where a student model trained on imperfect labels generated by a weaker teacher nonetheless surpasses that teacher, has been widely observed but the mechanisms that enable it have remained …
regressionNeural Collapse Inspired Knowledge Distillation
Existing knowledge distillation (KD) methods have demonstrated their ability in achieving student network performance on par with their teachers. However, the knowledge gap between the teacher and student remains signifi…
Knowledge DistillationBoomerang Distillation Enables Zero-Shot Model Size Interpolation
Large language models (LLMs) are typically deployed under diverse memory and compute constraints. Existing approaches build model families by training each size independently, which is prohibitively expensive and provide…
Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers
While feature-based knowledge distillation has proven highly effective for compressing CNNs, these techniques unexpectedly fail when applied to Vision Transformers (ViTs), often performing worse than simple logit-based d…
Knowledge Distillation