Adaptive Attention Span in Transformers

2019-05-19 · ACL 2019 7 · Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, Armand Joulin

We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.

📄 PDF Abstract BibTeX arXiv:1905.07799

Code (8)

facebookresearch/adaptive-span 공식 구현 pytorch

JoeRoussy/adaptive-attention-in-cv pytorch

jerrodparker20/adaptive-transformers-in-rl pytorch

lancopku/Explicit-Sparse-Transformer tf

ofirpress/sandwich_transformer pytorch

prajjwal1/adaptive_transformer pytorch

prajjwal1/fluence pytorch

pwc-1/Paper-9/tree/main/7/Knowing-When-to-Look-Adaptive-Attention mindspore

Tasks

8kLanguage ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

L1 Regularization $L_{1}$ Regularization is a regularization technique applied to the weights of a neural network. We minimize a loss function compromising both the primary loss function and a…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Embedding Dropout Embedding Dropout is equivalent to performing dropout on the embedding matrix at a word level, where the dropout is broadcast…

Adaptive Masking 설명 없음

Adaptive Span Transformer 설명 없음

Adaptive Transformers in RL

2020-04-08 · Shakti Kumar, Jerrod Parker, Panteha Naderian

Recent developments in Transformers have opened new interesting areas of research in partially observable reinforcement learning tasks. Results from late 2019 showed that Transformers are able to outperform LSTMs on both…

Partially Observable Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Adaptive Transformers for Learning Multimodal Representations

2020-05-15 · ACL 2020 6 · Prajjwal Bhargava

The usage of transformers has grown from learning about language semantics to forming meaningful visiolinguistic representations. These architectures are often over-parametrized, requiring large amounts of computation. I…

Computational Efficiency

Adaptive Attention Span in Computer Vision

2020-04-18 · Jerrod Parker, Shakti Kumar, Joe Roussy

Recent developments in Transformers for language modeling have opened new areas of research in computer vision. Results from late 2019 showed vast performance increases in both object detection and recognition when convo…

Language Modeling Language Modelling object-detection Object Detection

Cascaded Semantic and Positional Self-Attention Network for Document Classification

2020-09-15 · Findings of the Association for Computational Linguistics 2020 · Juyong Jiang, Jie Zhang, Kai Zhang

Transformers have shown great success in learning representations for language modelling. However, an open challenge still remains on how to systematically aggregate semantic information (word embedding) with positional …

Classification Document Classification General Classification Language Modelling

Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting

2024-02-04 · Peng Chen, Yingying Zhang, Yunyao Cheng, Yang Shu 외

Transformers for time series forecasting mainly model time series from limited or fixed scales, making it challenging to capture different characteristics spanning various scales. We propose Pathformer, a multi-scale Tra…

Time Series Time Series Forecasting