Deep Transformer Model with Pre-Layer Normalization for COVID-19 Growth Prediction

2022-07-10 · Rizki Ramadhan Fitra, Novanto Yudistira, Wayan Firdaus Mahmudy

Coronavirus disease or COVID-19 is an infectious disease caused by the SARS-CoV-2 virus. The first confirmed case caused by this virus was found at the end of December 2019 in Wuhan City, China. This case then spread throughout the world, including Indonesia. Therefore, the COVID-19 case was designated as a global pandemic by WHO. The growth of COVID-19 cases, especially in Indonesia, can be predicted using several approaches, such as the Deep Neural Network (DNN). One of the DNN models that can be used is Deep Transformer which can predict time series. The model is trained with several test scenarios to get the best model. The evaluation is finding the best hyperparameters. Then, further evaluation was carried out using the best hyperparameters setting of the number of prediction days, the optimizer, the number of features, and comparison with the former models of the Long Short-Term Memory (LSTM) and Recurrent Neural Network (RNN). All evaluations used metric of the Mean Absolute Percentage Error (MAPE). Based on the results of the evaluations, Deep Transformer produces the best results when using the Pre-Layer Normalization and predicting one day ahead with a MAPE value of 18.83. Furthermore, the model trained with the Adamax optimizer obtains the best performance among other tested optimizers. The performance of the Deep Transformer also exceeds other test models, which are LSTM and RNN.

📄 PDF Abstract BibTeX arXiv:2207.06356

Code (0)

등록된 구현이 없습니다.

Tasks

Time SeriesTime Series Analysis

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Test 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Stability of Transformers under Layer Normalization

2025-10-10 · Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai 외 arxiv

Despite their widespread use, training deep Transformers can be unstable. Layer normalization, a standard component, improves training stability, but its placement has often been ad-hoc. In this paper, we conduct a princ…

Methods of improving LLM training stability

2024-10-22 · Oleg Rybakov, Mike Chrzanowski, Peter Dykas, Jinze Xue 외

Training stability of large language models(LLMs) is an important research topic. Reproducing training instabilities can be costly, so we use a small language model with 830M parameters and experiment with higher learnin…

Small Language Model

COVID-19 growth prediction using multivariate long short term memory

2020-05-10 · Novanto Yudistira

Coronavirus disease (COVID-19) spread forecasting is an important task to track the growth of the pandemic. Existing predictions are merely based on qualitative analyses and mathematical modeling. The use of available bi…

Prediction

Peri-LN: Revisiting Layer Normalization in the Transformer Architecture

2025-02-04 · Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh 외

Designing Transformer architectures with the optimal layer normalization (LN) strategy that ensures large-scale training stability and expedite convergence has remained elusive, even in this era of large language models …

Subcritical Signal Propagation at Initialization in Normalization-Free Transformers

2026-04-13 · Sergey Alekseev arxiv

We study signal propagation at initialization in transformers through the averaged partial Jacobian norm (APJN), a measure of gradient amplification across layers. We extend APJN analysis to transformers with bidirection…