Semi-Supervised Online Learning on the Edge by Transforming Knowledge from Teacher Models
Edge machine learning (Edge ML) enables training ML models using the vast data distributed across network edges. However, many existing approaches assume static models trained centrally and then deployed, making them ineffective against unseen data. To address this, Online Edge ML allows models to be trained directly on edge devices and updated continuously with new data. This paper explores a key challenge of Online Edge ML: "How to determine labels for truly future, unseen data points". We propose Knowledge Transformation (KT), a hybrid method combining Knowledge Distillation, Active Learning, and causal reasoning. In short, KT acts as the oracle in active learning by transforming knowledge from a teacher model to generate pseudo-labels for training a student model. To verify the validity of the method, we conducted simulation experiments with two setups: (1) using a less stable teacher model and (2) a relatively more stable teacher model. Results indicate that when a stable teacher model is given, the student model can eventually reach its expected maximum performance. KT is potentially beneficial for scenarios that meet the following circumstances: (1) when the teacher's task is generic, which means existing pre-trained models might be adequate for its task, so there will be no need to train the teacher model from scratch; and/or (2) when the label for the student's task is difficult or expensive to acquire.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationActive LearningSimilar Papers 제목 키워드 기반
A rule-general abductive learning by rough sets
In real-world tasks, there is usually a large amount of unlabeled data and labeled data. The task of combining the two to learn is known as semi-supervised learning. Experts can use logical rules to label unlabeled data,…
Visual and Semantic Knowledge Transfer for Large Scale Semi-supervised Object Detection
Deep CNN-based object detection systems have achieved remarkable success on several large-scale object detection benchmarks. However, training such detectors requires a large number of labeled bounding boxes, which are m…
Objectobject-detectionObject DetectionSemi-Supervised Object Detection+1Large Scale Semi-Supervised Object Detection Using Visual and Semantic Knowledge Transfer
Deep CNN-based object detection systems have achieved remarkable success on several large-scale object detection benchmarks. However, training such detectors requires a large number of labeled bounding boxes, which are m…
Objectobject-detectionObject DetectionSemi-Supervised Object Detection+1Class Knowledge Overlay to Visual Feature Learning for Zero-Shot Image Classification
New categories can be discovered by transforming semantic features into synthesized visual features without corresponding training samples in zero-shot image classification. Although significant progress has been made in…
General Classificationimage-classificationImage ClassificationTriplet+2Knowledge Distillation for Road Detection based on cross-model Semi-Supervised Learning
The advancement of knowledge distillation has played a crucial role in enabling the transfer of knowledge from larger teacher models to smaller and more efficient student models, and is particularly beneficial for online…
Knowledge DistillationRoad Segmentation