Papers Self-Knowledge Distillation
“Self-Knowledge Distillation” 태그가 달린 논문 68편 · 필터 해제
X Modality Assisting RGBT Object Tracking
Learning robust multi-modal feature representations is critical for boosting tracking performance. To this end, we propose a novel X Modality Assisting Network (X-Net) to shed light on the impact of the fusion paradigm b…
Knowledge DistillationObjectObject TrackingOptical Flow Estimation+2Combining inherent knowledge of vision-language models with unsupervised domain adaptation through strong-weak guidance
Unsupervised domain adaptation (UDA) tries to overcome the tedious work of labeling data by leveraging a labeled source dataset and transferring its knowledge to a similar but different target dataset. Meanwhile, current…
Domain AdaptationKnowledge DistillationLanguage ModellingSelf-Knowledge Distillation+1Double Reverse Regularization Network Based on Self-Knowledge Distillation for SAR Object Classification
In current synthetic aperture radar (SAR) object classification, one of the major challenges is the severe overfitting issue due to the limited dataset (few-shot) and noisy data. Considering the advantages of knowledge d…
Knowledge DistillationSelf-Knowledge DistillationPromoting Generalized Cross-lingual Question Answering in Few-resource Scenarios via Self-knowledge Distillation
Despite substantial progress in multilingual extractive Question Answering (QA), models with high and uniformly distributed performance across languages remain challenging, especially for languages with limited resources…
Cross-Lingual Question AnsweringCross-Lingual TransferExtractive Question-AnsweringKnowledge Distillation+3FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning
Federated Learning (FL) aggregates locally trained models from individual clients to construct a global model. While FL enables learning a model with data privacy, it often suffers from significant performance degradatio…
Continual LearningFederated LearningImage ClassificationKnowledge Distillation+2Eyelid’s Intrinsic Motion-aware Feature Learning for Real-time Eyeblink Detection in the Wild
Real-time eyeblink detection in the wild is a recently emerged challenging task that suffers from dramatic variations in face attribute, pose, illumination, camera view and distance, etc. One key issue is to well charact…
AttributeDescriptiveEyeblink detectionKnowledge Distillation+1Three Factors to Improve Out-of-Distribution Detection
In the problem of out-of-distribution (OOD) detection, the usage of auxiliary data as outlier data for fine-tuning has demonstrated encouraging performance. However, previous methods have suffered from a trade-off betwee…
Contrastive LearningKnowledge DistillationOut-of-Distribution DetectionOut of Distribution (OOD) Detection+1Effective Whole-body Pose Estimation with Two-stages Distillation
Whole-body pose estimation localizes the human body, hand, face, and foot keypoints in an image. This task is challenging due to multi-scale body parts, fine-grained localization for low-resolution regions, and data scar…
2D Human Pose EstimationKnowledge DistillationPose EstimationSelf-Knowledge DistillationRobust Spatiotemporal Traffic Forecasting with Reinforced Dynamic Adversarial Training
Machine learning-based forecasting models are commonly used in Intelligent Transportation Systems (ITS) to predict traffic patterns and provide city-wide services. However, most of the existing models are susceptible to …
Adversarial RobustnessKnowledge DistillationSelf-Knowledge DistillationIncorporating Graph Information in Transformer-based AMR Parsing
Abstract Meaning Representation (AMR) is a Semantic Parsing formalism that aims at providing a semantic graph abstraction representing a given text. Current approaches are based on autoregressive language models such as …
Abstract Meaning RepresentationAMR ParsingKnowledge DistillationSelf-Knowledge Distillation+2Self-Knowledge Distillation for Surgical Phase Recognition
Purpose: Advances in surgical phase recognition are generally led by training deeper networks. Rather than going further with a more complex solution, we believe that current models can be exploited better. We propose a …
DecoderKnowledge DistillationSelf-Knowledge DistillationSurgical phase recognitionLightweight Self-Knowledge Distillation with Multi-source Information Fusion
Knowledge Distillation (KD) is a powerful technique for transferring knowledge between neural network models, where a pre-trained teacher model is used to facilitate the training of the target student model. However, the…
Knowledge DistillationSelf-Knowledge DistillationFrom Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels
Knowledge Distillation (KD) uses the teacher's prediction logits as soft labels to guide the student, while self-KD does not need a real teacher to require the soft labels. This work unifies the formulations of the two t…
Knowledge DistillationSelf-Knowledge DistillationConfidence Attention and Generalization Enhanced Distillation for Continuous Video Domain Adaptation
Continuous Video Domain Adaptation (CVDA) is a scenario where a source model is required to adapt to a series of individually available changing target domains continuously without source data or target supervision. It h…
Autonomous DrivingDomain AdaptationKnowledge DistillationSelf-Knowledge Distillation+1DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervision
Algorithmic fairness has become an important machine learning problem, especially for mission-critical Web applications. This work presents a self-supervised model, called DualFair, that can debias sensitive attributes l…
counterfactualFairnessKnowledge DistillationRepresentation Learning+1Graph-based Knowledge Distillation: A survey and experimental evaluation
Graph, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph Neural Networks (GNNs) have gained widespread attention for their robust expressiveness and exception…
Knowledge DistillationSelf-Knowledge DistillationSurveyYou Do Not Need Additional Priors or Regularizers in Retinex-Based Low-Light Image Enhancement
Images captured in low-light conditions often suffer from significant quality degradation. Recent works have built a large variety of deep Retinex-based networks to enhance low-light images. The Retinex-based methods…
Contrastive LearningImage EnhancementKnowledge DistillationLow-Light Image Enhancement+1Siamese Sleep Transformer For Robust Sleep Stage Scoring With Self-knowledge Distillation and Selective Batch Sampling
In this paper, we propose a Siamese sleep transformer (SST) that effectively extracts features from single-channel raw electroencephalogram signals for robust sleep stage scoring. Despite the significant advances in slee…
Knowledge DistillationSelf-Knowledge DistillationAI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation
We present a novel adversarial penalized self-knowledge distillation method, named adversarial learning and implicit regularization for self-knowledge distillation (AI-KD), which regularizes the training procedure by adv…
Knowledge DistillationSelf-Knowledge DistillationMultimodality Multi-Lead ECG Arrhythmia Classification using Self-Supervised Learning
Electrocardiogram (ECG) signal is one of the most effective sources of information mainly employed for the diagnosis and prediction of cardiovascular diseases (CVDs) connected with the abnormalities in heart rhythm. Clea…
ECG ClassificationKnowledge DistillationRhythmSelf-Knowledge Distillation+3