Training Very Deep Networks
Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.
Code (3)
Tasks
Image ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs
Large language models (LLMs) have made a profound impact across various fields due to their advanced capabilities. However, training these models at unprecedented scales requires extensive AI accelerator clusters and sop…
CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
The paper proposes and optimizes a partial recovery training system, CPR, for recommendation models. CPR relaxes the consistency requirement by enabling non-failed nodes to proceed without loading checkpoints when a node…
EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration
Robust embodied robots should be able to recover from failures and retry tasks in order to operate reliably in unstructured and noisy real-world environments. Achieving this capability requires training policies on data …
Does YOLO Really Need to See Every Training Image in Every Epoch?
YOLO detectors are known for their fast inference speed, yet training them remains unexpectedly time-consuming due to their exhaustive pipeline that processes every training image in every epoch, even when many images ha…
CRRL: A Causality-Based Reinforcement Learning Framework for Autonomous System Recovery
Traditional reinforcement learning (RL) for recovery in autonomous systems lacks causal understanding and generalizes poorly to novel failure scenarios. RL policies often stall in failure states, spending up to 70% of an…
Reinforcement Learning