Transformer-XL

2000년 도입 · 논문 64편에서 사용

Transformer-XL (meaning extra long) is a Transformer architecture that introduces the notion of recurrence to the deep self-attention network. Instead of computing the hidden states from scratch for each new segment, Transformer-XL reuses the hidden states obtained in previous segments. The reused hidden states serve as memory for the current segment, which builds up a recurrent connection between the segments. As a result, modeling very long-term dependency becomes possible because information can be propagated through the recurrent connections. As an additional contribution, the Transformer-XL uses a new relative positional encoding formulation that generalizes to attention lengths longer than the one observed during training.

출처: Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

소개 논문: Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context