Training Deep Networks from Zero to Hero: avoiding pitfalls and going beyond
Training deep neural networks may be challenging in real world data. Using models as black-boxes, even with transfer learning, can result in poor generalization or inconclusive results when it comes to small datasets or specific applications. This tutorial covers the basic steps as well as more recent options to improve models, in particular, but not restricted to, supervised learning. It can be particularly useful in datasets that are not as well-prepared as those in challenges, and also under scarce annotation and/or small data. We describe basic procedures: as data preparation, optimization and transfer learning, but also recent architectural choices such as use of transformer modules, alternative convolutional layers, activation functions, wide and deep networks, as well as training procedures including as curriculum, contrastive and self-supervised learning.
Code (1)
Tasks
Self-Supervised LearningTransfer LearningSimilar Papers 제목 키워드 기반
From Zero to Hero: Convincing with Extremely Complicated Math
Becoming a (super) hero is almost every kid's dream. During their sheltered childhood, they do whatever it takes to grow up to be one. Work hard, play hard -- all day long. But as they're getting older, distractions are …
MathLean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning
Split Federated Learning (SFL) enables collaborative training between resource-constrained edge devices and a compute-rich server. Communication overhead is a central issue in SFL and can be mitigated with auxiliary netw…
Federated LearningFrom SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents
We introduce SWE-ZERO to SWE-HERO, a two-stage SFT recipe that achieves state-of-the-art results on SWE-bench by distilling open-weight frontier LLMs. Our pipeline replaces resource-heavy dependencies with an evolutionar…
Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data
This paper shows how to use large-scale pre-trained language models to extract character roles from narrative texts without training data. Queried with a zero-shot question-answering prompt, GPT-3 can identify the hero, …
ArticlesQuestion AnsweringZeroQuant-HERO: Hardware-Enhanced Robust Optimized Post-Training Quantization Framework for W8A8 Transformers
Quantization techniques are pivotal in reducing the memory and computational demands of deep neural network inference. Existing solutions, such as ZeroQuant, offer dynamic quantization for models like BERT and GPT but ov…
Quantization