A Survey on Efficient Training of Transformers
Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.
Code (0)
등록된 구현이 없습니다.
Tasks
SurveyMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Foundation Models and Transformers for Anomaly Detection: A Survey
In line with the development of deep learning, this survey examines the transformative role of Transformers and foundation models in advancing visual anomaly detection (VAD). We explore how these architectures, with thei…
Anomaly DetectionLarge Language Models Meet Computer Vision: A Brief Survey
Recently, the intersection of Large Language Models (LLMs) and Computer Vision (CV) has emerged as a pivotal area of research, driving significant advancements in the field of Artificial Intelligence (AI). As transformer…
SurveyA survey of the Vision Transformers and their CNN-Transformer based Variants
Vision transformers have become popular as a possible substitute to convolutional neural networks (CNNs) for a variety of computer vision applications. These transformers, with their ability to focus on global relationsh…
SurveyVideo Transformers: A Survey
Transformer models have shown great success handling long-range interactions, making them a promising tool for modeling video. However, they lack inductive biases and scale quadratically with input length. These limitati…
Action ClassificationSelf-Supervised LearningSurveyTransformers in Remote Sensing: A Survey
Deep learning-based algorithms have seen a massive popularity in different areas of remote sensing image analysis over the past decade. Recently, transformers-based architectures, originally introduced in natural languag…
Survey