A Length-Extrapolatable Transformer
Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at https://aka.ms/LeX-Transformer.
Code (5)
Tasks
Language ModelingLanguage ModellingPositionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding
Diffusion transformers(DiTs) struggle to generate images at resolutions higher than their training resolutions. The primary obstacle is that the explicit positional encodings(PE), such as RoPE, need extrapolation which d…
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative positional embedding design, ALiBi, has had the …
Language ModelingLanguage ModellingAttention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
An ideal length-extrapolatable Transformer language model can handle sequences longer than the training length without any fine-tuning. Such long-context utilization capability relies heavily on a flexible positional emb…
Code CompletionLanguage ModelingLanguage ModellingQuestion Answering+1Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding
Built upon the Transformer, large language models (LLMs) have captured worldwide attention due to their remarkable abilities. Nevertheless, all Transformer-based models including LLMs suffer from a preset length limit an…
PositionSurveyTimelyGPT: Extrapolatable Transformer Pre-training for Long-term Time-Series Forecasting in Healthcare
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success in Natural Language Processing and Computer Vision domains. However, the development of PTMs on healthcare time-series data …
DiagnosticTime SeriesTime Series ForecastingTrajectory Prediction