Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
Transfer learning seeks to improve the generalization performance of a target task by exploiting the knowledge learned from a related source task. Central questions include deciding what information one should transfer and when transfer can be beneficial. The latter question is related to the so-called negative transfer phenomenon, where the transferred source information actually reduces the generalization performance of the target task. This happens when the two tasks are sufficiently dissimilar. In this paper, we present a theoretical analysis of transfer learning by studying a pair of related perceptron learning tasks. Despite the simplicity of our model, it reproduces several key phenomena observed in practice. Specifically, our asymptotic analysis reveals a phase transition from negative transfer to positive transfer as the similarity of the two tasks moves past a well-defined threshold.
Code (0)
등록된 구현이 없습니다.
Tasks
Transfer LearningVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
Unveiling phase transitions with machine learning
The classification of phase transitions is a central and challenging task in condensed matter physics. Typically, it relies on the identification of order parameters and the analysis of singularities in the free energy a…
BIG-bench Machine LearningTransfer LearningRethinking Breiman's Dilemma in Neural Networks: Phase Transitions of Margin Dynamics
Margin enlargement over training data has been an important strategy since perceptrons in machine learning for the purpose of boosting the robustness of classifiers toward a good generalization ability. Yet Breiman (1999…
Generalization BoundsTransfer learning of phase transitions in percolation and directed percolation
The latest advances of statistical physics have shown remarkable performance of machine learning in identifying phase transitions. In this paper, we apply domain adversarial neural network (DANN) based on transfer learni…
Transfer LearningNeural models for prediction of spatially patterned phase transitions: methods and challenges
Dryland vegetation ecosystems are known to be susceptible to critical transitions between alternative stable states when subjected to external forcing. Such transitions are often discussed through the framework of bifurc…
Detecting phase transitions in collective behavior using manifold's curvature
If a given behavior of a multi-agent system restricts the phase variable to a invariant manifold, then we define a phase transition as change of physical characteristics such as speed, coordination, and structure. We def…