Churn Reduction via Distillation
In real-world systems, models are frequently updated as more data becomes available, and in addition to achieving high accuracy, the goal is to also maintain a low difference in predictions compared to the base model (i.e. predictive "churn"). If model retraining results in vastly different behavior, then it could cause negative effects in downstream systems, especially if this churn can be avoided with limited impact on model accuracy. In this paper, we show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines on a wide range of datasets and model architectures, including fully-connected networks, convolutional networks, and transformers.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Reducing Model Churn: Stable Re-training of Conversational Agents
Retraining modern deep learning systems can lead to variations in model performance even when trained using the same data and hyper-parameters by simply using different random seeds. This phenomenon is known as model chu…
Semantic ParsingStructured PredictionOn the Reproducibility of Neural Network Predictions
Standard training techniques for neural networks involve multiple sources of randomness, e.g., initialization, mini-batch ordering and in some cases data augmentation. Given that neural networks are heavily over-paramete…
Data Augmentationimage-classificationImage ClassificationMitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn:…
Continual LearningOpenAI GymChurn Identification in Microblogs using Convolutional Neural Networks with Structured Logical Knowledge
For brands, gaining new customer is more expensive than keeping an existing one. Therefore, the ability to keep customers in a brand is becoming more challenging these days. Churn happens when a customer leaves a brand t…
Language ModelingLanguage ModellingMachine TranslationSentiment Analysis+1Maintaining Stability and Plasticity for Predictive Churn Reduction
Deployed machine learning models should be updated to take advantage of a larger sample size to improve performance, as more data is gathered over time. Unfortunately, even when model updates improve aggregate metrics su…