Distilling Critical Paths in Convolutional Neural Networks
Neural network compression and acceleration are widely demanded currently due to the resource constraints on most deployment targets. In this paper, through analyzing the filter activation, gradients, and visualizing the filters' functionality in convolutional neural networks, we show that the filters in higher layers learn extremely task-specific features, which are exclusive for only a small subset of the overall tasks, or even a single class. Based on such findings, we reveal the critical paths of information flow for different classes. And by their intrinsic property of exclusiveness, we propose a critical path distillation method, which can effectively customize the convolutional neural networks to small ones with much smaller model size and less computation.
Code (0)
등록된 구현이 없습니다.
Tasks
Neural Network CompressionSimilar Papers 제목 키워드 기반
System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts
Chain-of-thought (CoT) reasoning enables large language models (LLMs) to move beyond fast System-1 responses and engage in deliberative System-2 reasoning. However, this comes at the cost of significant inefficiency due …
GSM8KAdaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
Knowledge Distillation is becoming one of the primary trends among neural network compression algorithms to improve the generalization performance of a smaller student model with guidance from a larger teacher model. Thi…
Knowledge DistillationNeural Network Compressionobject-detectionObject Detection+1[Re] Distilling Knowledge via Knowledge Review
This effort aims to reproduce the results of experiments and analyze the robustness of the review framework for knowledge distillation introduced in the CVPR '21 paper 'Distilling Knowledge via Knowledge Review' by Chen …
HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification
Vision Transformers (ViTs) have achieved significant advancement in computer vision tasks due to their powerful modeling capacity. However, their performance notably degrades when trained with insufficient data due to la…
Computational Efficiencyimage-classificationImage ClassificationKnowledge Distillation+1Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
One-shot weight sharing methods have recently drawn great attention in neural architecture search due to high efficiency and competitive performance. However, weight sharing across models has an inherent deficiency, i.e.…
Neural Architecture Searchobject-detectionObject Detection