An information-Theoretic Approach to Semi-supervised Transfer Learning
Transfer learning is a valuable tool in deep learning as it allows propagating information from one "source dataset" to another "target dataset", especially in the case of a small number of training examples in the latter. Yet, discrepancies between the underlying distributions of the source and target data are commonplace and are known to have a substantial impact on algorithm performance. In this work we suggest novel information-theoretic approaches for the analysis of the performance of deep neural networks in the context of transfer learning. We focus on the task of semi-supervised transfer learning, in which unlabeled samples from the target dataset are available during network training on the source dataset. Our theory suggests that one may improve the transferability of a deep neural network by incorporating regularization terms on the target data based on information-theoretic quantities, namely the Mutual Information and the Lautum Information. We demonstrate the effectiveness of the proposed approaches in various semi-supervised transfer learning experiments.
Code (0)
등록된 구현이 없습니다.
Tasks
Transfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Lautum Regularization for Semi-supervised Transfer Learning
Transfer learning is a very important tool in deep learning as it allows propagating information from one "source dataset" to another "target dataset", especially in the case of a small number of training examples in the…
Transfer LearningCo-regularization Based Semi-supervised Domain Adaptation
This paper presents a co-regularization based approach to semi-supervised domain adaptation. Our proposed approach (EA++) builds on the notion of augmented space (introduced in EASYADAPT (EA) [1]) and harnesses unlabeled…
Domain AdaptationGeneralization BoundsSemi-supervised Domain AdaptationSentiment AnalysisSemi-Supervised Clustering via Information-Theoretic Markov Chain Aggregation
We connect the problem of semi-supervised clustering to constrained Markov aggregation, i.e., the task of partitioning the state space of a Markov chain. We achieve this connection by considering every data point in the …
ClusteringAdversarial Training Helps Transfer Learning via Better Representations
Transfer learning aims to leverage models pre-trained on source data to efficiently adapt to target setting, where only limited data are available for model fine-tuning. Recent works empirically demonstrate that adversar…
Transfer LearningTransMedSeg: A Transferable Semantic Framework for Semi-Supervised Medical Image Segmentation
Semi-supervised learning (SSL) has achieved significant progress in medical image segmentation (SSMIS) through effective utilization of limited labeled data. While current SSL methods for medical images predominantly rel…
Image SegmentationMedical Image AnalysisMedical Image SegmentationRepresentation Learning+2