SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis

2024-12-26 · Huiyuan Tian, Bonan Xu, Shijian Li, Gang Pan

Knowledge Distillation (KD) has achieved widespread success in compressing large Vision Transformers (ViTs), but a unified theoretical framework for both ViTs and KD is still lacking. In this paper, we propose SpectralKD, a novel unified analytical framework that offers deeper insights into ViTs and optimizes KD via spectral analysis. Our model-wise analysis reveals that CaiT concentrates information in their first and last few layers, informing optimal layer selection for KD. Surprisingly, our layer-wise analysis discovers that Swin Transformer and CaiT exhibit similar spectral encoding patterns despite their architectural differences, leading to feature map alignment guideline. Building on these insights, we propose a simple yet effective spectral alignment method for KD. Benefiting from the deeper understanding by above analysis results, even such a simple strategy achieves state-of-the-art performance on ImageNet-1K without introducing any trainable parameters, improving DeiT-Tiny by $+5.2\%$ and Swin-Tiny by $+1.4\%$ in top-1 accuracy. Furthermore, our post-training analysis reveals that distilled students can reproduce spectral patterns similar to their teachers, opening a new area we term ``distillation dynamics". Code and experimental logs are available in https://github.com/thy960112/SpectralKD.

📄 PDF Abstract BibTeX arXiv:2412.19055

Code (1)

thy960112/SpectralKD 공식 구현 pytorch

Tasks

Knowledge Distillation Transfer Learning

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Stochastic Depth Stochastic Depth aims to shrink the depth of a network during training, while keeping it unchanged during testing. This is achieved by randomly dropping entire…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Class Attention A Class Attention layer, or CA Layer, is an attention mechanism for [vision…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Interpreting Neural Combinatorial Optimization via Evolving Programmatic Bottlenecks

2026-06-18 · Haocheng Duan, Yuxin Guo, Jieyi Bi, Anqi Xie 외 arxiv

Neural Combinatorial Optimization (NCO) achieves strong performance, yet its black-box nature remains a key roadblock to deployment and scientific diagnosis. Standard interpretability tools, such as Concept Bottleneck Mo…

VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set

2025-10-24 · Shufan Shen, Junshu Sun, Qingming Huang, Shuhui Wang arxiv

The alignment of vision-language representations endows current Vision-Language Models (VLMs) with strong multi-modal reasoning capabilities. However, the interpretability of the alignment component remains uninvestigate…

Zero-Shot Image Classification Semantic Similarity

TSInterpret: A unified framework for time series interpretability

2022-08-10 · Jacqueline Höllig, Cedric Kulbach, Steffen Thoma

With the increasing application of deep learning algorithms to time series classification, especially in high-stake scenarios, the relevance of interpreting those algorithms becomes key. Although research in time series …

Interpretable Machine Learning Time Series Time Series Analysis Time Series Classification

Distilling Inductive Bias: Knowledge Distillation Beyond Model Compression

2023-09-30 · Gousia Habib, Tausifa Jan Saleem, Brejesh lall

With the rapid development of computer vision, Vision Transformers (ViTs) offer the tantalizing prospect of unified information processing across visual and textual domains. But due to the lack of inherent inductive bias…

Inductive Bias Knowledge Distillation model Model Compression

UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

2026-06-18 · Wenhao Chi, Arkaprava Sinha, Dominick Reilly, Hieu Le 외 arxiv

Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. We argue that a …

Representation Learning Action Segmentation Action Recognition Video Retrieval