QuickNets: Saving Training and Preventing Overconfidence in Early-Exit Neural Architectures
Deep neural networks have long training and processing times. Early exits added to neural networks allow the network to make early predictions using intermediate activations in the network in time-sensitive applications. However, early exits increase the training time of the neural networks. We introduce QuickNets: a novel cascaded training algorithm for faster training of neural networks. QuickNets are trained in a layer-wise manner such that each successive layer is only trained on samples that could not be correctly classified by the previous layers. We demonstrate that QuickNets can dynamically distribute learning and have a reduced training cost and inference cost compared to standard Backpropagation. Additionally, we introduce commitment layers that significantly improve the early exits by identifying for over-confident predictions and demonstrate its success.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Uncertainty-Estimation with Normalized Logits for Out-of-Distribution Detection
Out-of-distribution (OOD) detection is critical for preventing deep learning models from making incorrect predictions to ensure the safety of artificial intelligence systems. Especially in safety-critical applications su…
Autonomous DrivingMedical DiagnosisOut-of-Distribution DetectionOut of Distribution (OOD) DetectionPrevent the Language Model from being Overconfident in Neural Machine Translation
The Neural Machine Translation (NMT) model is essentially a joint language model conditioned on both the source sentence and partial translation. Therefore, the NMT model naturally involves the mechanism of the Language …
HallucinationLanguage ModelingLanguage ModellingMachine Translation+3MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
We introduce MoMa, a novel modality-aware mixture-of-experts (MoE) architecture designed for pre-training mixed-modal, early-fusion language models. MoMa processes images and text in arbitrary sequences by dividing exper…
Causal InferenceLanguage ModellingMixture-of-ExpertsgSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling
A large catalogue size is one of the central challenges in training recommendation models: a large number of items makes them memory and computationally inefficient to compute scores for all items during training, forcin…
Sequential RecommendationRethinking Early Stopping: Refine, Then Calibrate
Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with prope…
Decision Making