RealFormer

2000년 도입 · 논문 1편에서 사용

RealFormer is a type of Transformer based on the idea of residual attention. It adds skip edges to the backbone Transformer to create multiple direct paths, one for each type of attention module. It adds no parameters or hyper-parameters. Specifically, RealFormer uses a Post-LN style Transformer as backbone and adds skip edges to connect Multi-Head Attention modules in adjacent layers.

출처: RealFormer: Transformer Likes Residual Attention

소개 논문: RealFormer: Transformer Likes Residual Attention