Using Pre-trained LLMs for Multivariate Time Series Forecasting

2025-01-10 · Malcolm L. Wolff, Shenghao Yang, Kari Torkkola, Michael W. Mahoney

Pre-trained Large Language Models (LLMs) encapsulate large amounts of knowledge and take enormous amounts of compute to train. We make use of this resource, together with the observation that LLMs are able to transfer knowledge and performance from one domain or even modality to another seemingly-unrelated area, to help with multivariate demand time series forecasting. Attention in transformer-based methods requires something worth attending to -- more than just samples of a time-series. We explore different methods to map multivariate input time series into the LLM token embedding space. In particular, our novel multivariate patching strategy to embed time series features into decoder-only pre-trained Transformers produces results competitive with state-of-the-art time series forecasting models. We also use recently-developed weight-based diagnostics to validate our findings.

📄 PDF Abstract BibTeX arXiv:2501.06386

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Multivariate Time Series ForecastingTime SeriesTime Series Forecasting

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Patching Activation patching studies the model's computation by altering its latent representations, the token embeddings in transformer-based language models, during the inference process

Causal Graph Fuzzy LLMs: A First Introduction and Applications in Time Series Forecasting

2025-07-22 · Omid Orang, Patricia O. Lucas, Gabriel I. F. Paiva, Petronio C. L. Silva 외 arxiv

In recent years, the application of Large Language Models (LLMs) to time series forecasting (TSF) has garnered significant attention among researchers. This study presents a new frame of LLMs named CGF-LLM using GPT-2 co…

Time Series Forecasting

MultiCast: Zero-Shot Multivariate Time Series Forecasting Using LLMs

2024-05-23 · Georgios Chatzigeorgakidis, Konstantinos Lentzos, Dimitrios Skoutas

Predicting future values in multivariate time series is vital across various domains. This work explores the use of large language models (LLMs) for this task. However, LLMs typically handle one-dimensional data. We intr…

Multivariate Time Series Forecasting Quantization Time Series Time Series Forecasting

LLM-Mixer: Multiscale Mixing in LLMs for Time Series Forecasting

2024-10-15 · Md Kowsher, Md. Shohanur Islam Sobuj, Nusrat Jahan Prottasha, E. Alejandro Alanis 외

Time series forecasting remains a challenging task, particularly in the context of complex multiscale temporal patterns. This study presents LLM-Mixer, a framework that improves forecasting accuracy through the combinati…

Time Series Time Series Forecasting

LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters

2023-08-16 · Ching Chang, Wei-Yao Wang, Wen-Chih Peng, Tien-Fu Chen

Multivariate time-series forecasting is vital in various domains, e.g., economic planning and weather prediction. Deep train-from-scratch models have exhibited effective performance yet require large amounts of data, whi…

Chatbot Multivariate Time Series Forecasting Representation Learning Self-Supervised Learning+3

Only the Curve Shape Matters: Training Foundation Models for Zero-Shot Multivariate Time Series Forecasting through Next Curve Shape Prediction

2024-02-12 · Cheng Feng, Long Huang, Denis Krompass

We present General Time Transformer (GTT), an encoder-only style foundation model for zero-shot multivariate time series forecasting. GTT is pretrained on a large dataset of 200M high-quality time series samples spanning…

Multivariate Time Series Forecasting Time Series Time Series Forecasting