paper-with-me

홈 › Papers

Sparse Low-Ranked Self-Attention Transformer for Remaining Useful Lifetime Prediction of Optical Fiber Amplifiers

2024-09-22 · Dominic Schneider, Lutz Rapp

Optical fiber amplifiers are key elements in present optical networks. Failures of these components result in high financial loss of income of the network operator as the communication traffic over an affected link is interrupted. Applying Remaining useful lifetime (RUL) prediction in the context of Predictive Maintenance (PdM) to optical fiber amplifiers to predict upcoming system failures at an early stage, so that network outages can be minimized through planning of targeted maintenance actions, ensures reliability and safety. Optical fiber amplifier are complex systems, that work under various operating conditions, which makes correct forecasting a difficult task. Increased monitoring capabilities of systems results in datasets that facilitate the application of data-driven RUL prediction methods. Deep learning models in particular have shown good performance, but generalization based on comparatively small datasets for RUL prediction is difficult. In this paper, we propose Sparse Low-ranked self-Attention Transformer (SLAT) as a novel RUL prediction method. SLAT is based on an encoder-decoder architecture, wherein two parallel working encoders extract features for sensors and time steps. By utilizing the self-attention mechanism, long-term dependencies can be learned from long sequences. The implementation of sparsity in the attention matrix and a low-rank parametrization reduce overfitting and increase generalization. Experimental application to optical fiber amplifiers exemplified on EDFA, as well as a reference dataset from turbofan engines, shows that SLAT outperforms the state-of-the-art methods.

📄 PDF Abstract BibTeX arXiv:2409.14378

Code (0)

등록된 구현이 없습니다.

Tasks

Prediction

Methods 이 논문이 사용한 방법론

Attention 설명 없음
Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
Position-Wise Feed-Forward Layer 설명 없음
Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…
BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…
Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Similar Papers 제목 키워드 기반

ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design

2022-10-18 · Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu 외

Vision Transformers (ViTs) have achieved state-of-the-art performance on various vision tasks. However, ViTs' self-attention module is still arguably a major bottleneck, limiting their achievable hardware efficiency. Mea…

Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models

2026-08-19 · Pardis Taghavi, Reza Langari, Gaurav Pandey arxiv

Training-free block-sparse attention can accelerate video transformers, but row-wise attention concentration does not by itself specify an executable sparse operator. Queries sharing a block route may have poorly overlap…

Video Generation

Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition

2022-09-30 · Chendong Zhao, Jianzong Wang, Wen qi Wei, Xiaoyang Qu 외

The Transformer architecture model, based on self-attention and multi-head attention, has achieved remarkable success in offline end-to-end Automatic Speech Recognition (ASR). However, self-attention and multi-head atten…

Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech Recognition

Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection

2019-12-25 · Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren 외

Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the ext…

Image CaptioningLanguage ModelingLanguage ModellingMachine Translation+1

Long Document Ranking with Query-Directed Sparse Transformer

2020-10-23 · Findings of the Association for Computational Linguistics 2020 · Jyun-Yu Jiang, Chenyan Xiong, Chia-Jung Lee, Wei Wang

The computing cost of transformer self-attention often necessitates breaking long documents to fit in pretrained models in document ranking tasks. In this paper, we design Query-Directed Sparse attention that induces IR-…

Document Ranking