A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting
The current landscape in time-series forecasting is dominated by Transformer-based models. Their high parameter count and corresponding demand in computational resources pose a challenge to real-world deployment, especially for commercial and scientific applications with low-power embedded devices. Pruning is an established approach to reduce neural network parameter count and save compute. However, the implications and benefits of pruning Transformer-based models for time series forecasting are largely unknown. To close this gap, we provide a comparative benchmark study by evaluating unstructured and structured pruning on various state-of-the-art multivariate time series models. We study the effects of these pruning strategies on model predictive performance and computational aspects like model size, operations, and inference time. Our results show that certain models can be pruned even up to high sparsity levels, outperforming their dense counterpart. However, fine-tuning pruned models is necessary. Furthermore, we demonstrate that even with corresponding hardware and software support, structured pruning is unable to provide significant time savings.
Code (0)
등록된 구현이 없습니다.
Tasks
Time SeriesTime Series ForecastingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On Designing Light-Weight Object Trackers through Network Pruning: Use CNNs or Transformers?
Object trackers deployed on low-power devices need to be light-weight, however, most of the current state-of-the-art (SOTA) methods rely on using compute-heavy backbones built using CNNs or transformers. Large sizes of s…
Network PruningObjectObject TrackingToken Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
Token compression techniques have recently emerged as powerful tools for accelerating Vision Transformer (ViT) inference in computer vision. Due to the quadratic computational complexity with respect to the token sequenc…
AI AgentEarly Transformers: A study on Efficient Training of Transformer Models through Early-Bird Lottery Tickets
The training of Transformer models has revolutionized natural language processing and computer vision, but it remains a resource-intensive and time-consuming process. This paper investigates the applicability of the earl…
Energy-Efficient Transformer Inference: Optimization Strategies for Time Series Classification
The increasing computational demands of transformer models in time series classification necessitate effective optimization strategies for energy-efficient deployment. Our study presents a systematic investigation of opt…
ClassificationInference OptimizationQuantizationTime Series+1Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective
While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, whic…