paper-with-me

홈 › Papers

A Unified Hyperparameter Optimization Pipeline for Transformer-Based Time Series Forecasting Models

2025-01-02 · Jingjing Xu, Caesar Wu, Yuan-Fang Li, Grégoire Danoy, Pascal Bouvry

Transformer-based models for time series forecasting (TSF) have attracted significant attention in recent years due to their effectiveness and versatility. However, these models often require extensive hyperparameter optimization (HPO) to achieve the best possible performance, and a unified pipeline for HPO in transformer-based TSF remains lacking. In this paper, we present one such pipeline and conduct extensive experiments on several state-of-the-art (SOTA) transformer-based TSF models. These experiments are conducted on standard benchmark datasets to evaluate and compare the performance of different models, generating practical insights and examples. Our pipeline is generalizable beyond transformer-based architectures and can be applied to other SOTA models, such as Mamba and TimeMixer, as demonstrated in our experiments. The goal of this work is to provide valuable guidance to both industry practitioners and academic researchers in efficiently identifying optimal hyperparameters suited to their specific domain applications. The code and complete experimental results are available on GitHub.

📄 PDF Abstract BibTeX arXiv:2501.01394

Code (1)

jingjing-unilu/HPO_transformer_time_series 공식 구현 pytorch

Tasks

Hyperparameter OptimizationMambaTime SeriesTime Series Forecasting

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Attention 설명 없음
HPO In machine learning, a hyperparameter is a parameter whose value is used to control learning process, and HPO is the problem of choosing a set of optimal hyperparameters for a…
Mamba Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.…

Similar Papers 제목 키워드 기반

FLASH: Fast Bayesian Optimization for Data Analytic Pipelines

2016-02-20 · Yuyu Zhang, Mohammad Taha Bahadori, Hang Su, Jimeng Sun

Modern data science relies on data analytic pipelines to organize interdependent computational steps. Such analytic pipelines often involve different algorithms across multiple steps, each with its own hyperparameters. T…

Bayesian Optimization

Reducing Hyperparameter Tuning Costs in ML, Vision and Language Model Training Pipelines via Memoization-Awareness

2024-11-06 · Abdelmajid Essofi, Ridwan Salahuddeen, Munachiso Nwadike, Elnura Zhalieva 외

The training or fine-tuning of machine learning, vision, and language models is often implemented as a pipeline: a sequence of stages encompassing data preparation, model training and evaluation. In this paper, we exploi…

Bayesian OptimizationGPULanguage ModelingLanguage Modelling

Be aware of overfitting by hyperparameter optimization!

2024-07-30 · Igor V. Tetko, Ruud Van Deursen, Guillaume Godin

Hyperparameter optimization is very frequently employed in machine learning. However, an optimization of a large space of parameters could result in overfitting of models. In recent studies on solubility prediction the a…

Hyperparameter OptimizationRepresentation Learning

Quantifying contribution and propagation of error from computational steps, algorithms and hyperparameter choices in image classification pipelines

2019-02-21 · Aritra Chowdhury, Malik Magdon-Ismail, Bulent Yener

Data science relies on pipelines that are organized in the form of interdependent computational steps. Each step consists of various candidate algorithms that maybe used for performing a particular function. Each algorit…

Bayesian OptimizationBIG-bench Machine LearningGeneral ClassificationHyperparameter Optimization+2

Learning To Exploit the Sequence-Specific Prior Knowledge for Image Processing Pipelines Optimization

2023-01-01 · CVPR 2023 1 · Haina Qin, Longfei Han, Weihua Xiong, Juan Wang 외

The hardware image signal processing (ISP) pipeline is the intermediate layer between the imaging sensor and the downstream application, processing the sensor signal into an RGB image. The ISP is less programmable an…

Hyperparameter OptimizationImage Segmentationobject-detectionObject Detection+1