Soft Knowledge Distillation with Multi-Dimensional Cross-Net Attention for Image Restoration Models Compression
Transformer-based encoder-decoder models have achieved remarkable success in image-to-image transfer tasks, particularly in image restoration. However, their high computational complexity-manifested in elevated FLOPs and parameter counts-limits their application in real-world scenarios. Existing knowledge distillation methods in image restoration typically employ lightweight student models that directly mimic the intermediate features and reconstruction results of the teacher, overlooking the implicit attention relationships between them. To address this, we propose a Soft Knowledge Distillation (SKD) strategy that incorporates a Multi-dimensional Cross-net Attention (MCA) mechanism for compressing image restoration models. This mechanism facilitates interaction between the student and teacher across both channel and spatial dimensions, enabling the student to implicitly learn the attention matrices. Additionally, we employ a Gaussian kernel function to measure the distance between student and teacher features in kernel space, ensuring stable and efficient feature learning. To further enhance the quality of reconstructed images, we replace the commonly used L1 or KL divergence loss with a contrastive learning loss at the image level. Experiments on three tasks-image deraining, deblurring, and denoising-demonstrate that our SKD strategy significantly reduces computational complexity while maintaining strong image restoration capabilities.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningDeblurringDecoderDenoisingImage RestorationKnowledge DistillationRain RemovalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning
Knowledge Graph Embedding (KGE) is a popular method for KG reasoning and training KGEs with higher dimension are usually preferred since they have better reasoning capability. However, high-dimensional KGEs pose huge cha…
Graph EmbeddingKnowledge DistillationKnowledge Graph EmbeddingKnowledge Graph Embeddings+1C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation
Existing Knowledge Distillation (KD) methods typically focus on transferring knowledge from a large-capacity teacher to a low-capacity student model achieving substantial success in unimodal knowledge transfer. Howev…
Knowledge DistillationTransfer LearningBalancing Knowledge Distillation for Imbalance Learning with Bilevel Optimization
Knowledge distillation transfers knowledge from a high capacity teacher to a compact student using a mixture of hard and soft losses. On imbalanced data, a fixed weighting between hard and soft losses becomes brittle the…
Knowledge DistillationBilevel OptimizationAligning Logits Generatively for Principled Black-Box Knowledge Distillation
Black-Box Knowledge Distillation (B2KD) is a formulated problem for cloud-to-edge model compression with invisible data and models hosted on the server. B2KD faces challenges such as limited Internet exchange and edge-cl…
Federated LearningKnowledge DistillationModel CompressionSelf-distillation with Batch Knowledge Ensembling Improves ImageNet Classification
The recent studies of knowledge distillation have discovered that ensembling the "dark knowledge" from multiple teachers or students contributes to creating better soft targets for training, but at the cost of significan…
ClassificationGeneral ClassificationKnowledge Distillation