Self Distillation via Iterative Constructive Perturbations
Deep Neural Networks have achieved remarkable achievements across various domains, however balancing performance and generalization still remains a challenge while training these networks. In this paper, we propose a novel framework that uses a cyclic optimization strategy to concurrently optimize the model and its input data for better training, rethinking the traditional training paradigm. Central to our approach is Iterative Constructive Perturbation (ICP), which leverages the model's loss to iteratively perturb the input, progressively constructing an enhanced representation over some refinement steps. This ICP input is then fed back into the model to produce improved intermediate features, which serve as a target in a self-distillation framework against the original features. By alternately altering the model's parameters to the data and the data to the model, our method effectively addresses the gap between fitting and generalization, leading to enhanced performance. Extensive experiments demonstrate that our approach not only mitigates common performance bottlenecks in neural networks but also demonstrates significant improvements across training variations.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Model Mimic Attack: Knowledge Distillation for Provably Transferable Adversarial Examples
The vulnerability of artificial neural networks to adversarial perturbations in the black-box setting is widely studied in the literature. The majority of attack methods to construct these perturbations suffer from an im…
Knowledge DistillationSpatial-then-Temporal Self-Supervised Learning for Video Correspondence
In low-level video analyses, effective representations are important to derive the correspondences between video frames. These representations have been learned in a self-supervised fashion from unlabeled images or video…
Contrastive LearningSelf-Supervised LearningIterative Self Knowledge Distillation -- From Pothole Classification to Fine-Grained and COVID Recognition
Pothole classification has become an important task for road inspection vehicles to save drivers from potential car accidents and repair bills. Given the limited computational power and fixed number of training epochs, w…
ClassificationKnowledge DistillationSelf-Knowledge DistillationSIKeD: Self-guided Iterative Knowledge Distillation for mathematical reasoning
Large Language Models (LLMs) can transfer their reasoning skills to smaller models by teaching them to generate the intermediate reasoning process required to solve multistep reasoning tasks. While LLMs can accurately so…
Knowledge DistillationMathematical ReasoningIterative Graph Self-Distillation
Recently, there has been increasing interest in the challenge of how to discriminatively vectorize graphs. To address this, we propose a method called Iterative Graph Self-Distillation (IGSD) which learns graph-level rep…
Contrastive LearningGraph LearningKnowledge Distillation